37: Unix Year 2038 problem and the art of underestimating

WEBVTT
00:00:00.000 --> 00:00:14.000
Hey everybody, it is another episode of Runtime Arguments. I'm Wolf, and as always, I'm with my very good best friend, Jim.
00:00:14.000 --> 00:00:15.000
Jim?
00:00:15.000 --> 00:00:17.000
Hello, Wolf, how you doing?
00:00:18.000 --> 00:00:19.000
You know.
00:00:19.000 --> 00:00:20.000
Pretty good.
00:00:20.000 --> 00:00:25.000
Um, this is episode number 37.
00:00:25.000 --> 00:00:30.000
Our 38th episode. I think I'm going to get tired of that math.
00:00:31.000 --> 00:00:37.000
Maybe. Um, and today, we are…
00:00:32.000 --> 00:00:35.000
Yeah, it's tough adding one.
00:00:36.000 --> 00:00:37.000
Hell yeah.
00:00:39.000 --> 00:00:42.000
Today we are going to talk about.
00:00:39.000 --> 00:00:40.000
Yeah.
00:00:42.000 --> 00:00:46.000
A class of bugs.
00:00:46.000 --> 00:00:47.000
Umm.
00:00:48.000 --> 00:00:51.000
That really should be avoidable.
00:00:51.000 --> 00:00:58.000
Um, I can't wait to give my opinion on this, because I absolutely have one! Um…
00:00:59.000 --> 00:01:00.000
Of course you do.
00:01:00.000 --> 00:01:11.000
And I think in about 6 minutes of Jim talking, you're gonna know what my opinion is, and I won't actually need to say it, but I'm gonna say it anyway.
00:01:10.000 --> 00:01:12.000
Yeah, okay.
00:01:11.000 --> 00:01:13.000
And.
00:01:13.000 --> 00:01:22.000
Let's start with feedback. I don't think we have any feedback for this show today. I didn't get any.
00:01:20.000 --> 00:01:34.000
No, we had a, you know, we had a lot of downloads on the, on the past episode, um, but, uh, we didn't, didn't get any feedback. Uh, uh, must be, uh, back to school, right? Everybody's busy.
00:01:29.000 --> 00:01:32.000
Which is interesting, it was a.
00:01:34.000 --> 00:01:39.000
It was a non-technical episode. Um, you know…
00:01:37.000 --> 00:01:40.000
Yeah, kind of abstract.
00:01:40.000 --> 00:01:45.000
You're very quiet, Jim. That can't be me. Is it me?
00:01:40.000 --> 00:01:42.000
A thinker.
00:01:42.000 --> 00:01:44.000
Am I?
00:01:44.000 --> 00:01:45.000
Is that better?
00:01:48.000 --> 00:01:49.000
I don't know.
00:01:49.000 --> 00:01:50.000
I don't know.
00:01:49.000 --> 00:01:59.000
Um… I hope people listened to the last episode because I.
00:01:59.000 --> 00:02:02.000
Felt like it was really important stuff.
00:02:02.000 --> 00:02:05.000
Um, that I wanted to say. But that means…
00:02:03.000 --> 00:02:06.000
Let's remind everybody what that was.
00:02:06.000 --> 00:02:18.000
Uh, we talked about leaning your ladder on the right wall, and uh, that what you really have to spend is not time, it's attention.
00:02:06.000 --> 00:02:08.000
What was that episode?
00:02:18.000 --> 00:02:29.000
Uh, and attention has different values, depending on the activity, and you have a different amount of attention.
00:02:29.000 --> 00:02:46.000
Based on your personal situation. Are you married with kids? Are you on vacation? A million. Are you tired? A million different things control just how much of this you have to spend.
00:02:41.000 --> 00:02:42.000
Yep.
00:02:46.000 --> 00:02:51.000
And attention is a thing you run out of every day.
00:02:49.000 --> 00:02:53.000
Yeah, it's a finite amount.
00:02:52.000 --> 00:03:06.000
It is, and maybe, maybe you have a lot. Maybe you have 6 hours worth of attention that you can put into stuff. Maybe you only have 4. Um, maybe you only have 2.
00:03:04.000 --> 00:03:05.000
Oh.
00:03:06.000 --> 00:03:08.000
Ah.
00:03:06.000 --> 00:03:09.000
Or maybe 15 minutes.
00:03:08.000 --> 00:03:13.000
Could be. I thought it was an important episode.
00:03:12.000 --> 00:03:13.000
It was good.
00:03:13.000 --> 00:03:19.000
Not too late for feedback. Everybody send your feedback. But this means more time for…
00:03:16.000 --> 00:03:18.000
Yeah, please.
00:03:19.000 --> 00:03:21.000
Jim, how was your week?
00:03:21.000 --> 00:03:36.000
Busy, as always. You know, I mentioned this a few months ago. I'm working on a project. This is something that's been brewing in my brain for 15 years, and I finally started it back in June, actually working on it.
00:03:36.000 --> 00:03:45.000
Uh, it's, it's this project that involves Node and TypeScript and lots of stuff that I, uh, I'm, I'm not.
00:03:45.000 --> 00:03:53.000
I'm not a real TypeScript programmer, so I'm learning TypeScript along the way, and I'm having Claude help me, and that's been a lot of fun.
00:03:53.000 --> 00:04:08.000
But this application that I've been working on, I'm going to show the customer in early October, and it's ready. But what I'm working on now is the documentation. So that's, that's the point I'm at. It's, it's functionally ready.
00:04:08.000 --> 00:04:22.000
And I can't wait to show them, because it's really gonna… gonna impress them, I think. And of course, I'll, I'll, uh, I'll let y'all know how that turns out. Um, so that's been, that's been consuming all of my attention lately.
00:04:22.000 --> 00:04:24.000
So that's been good.
00:04:24.000 --> 00:04:31.000
Uh, I'm super glad to hear you stepping out of your comfort zone and learning new stuff.
00:04:24.000 --> 00:04:27.000
Um, yeah.
00:04:31.000 --> 00:04:32.000
Yeah.
00:04:31.000 --> 00:04:39.000
Which you do. If I ever imply that you don't, that would be a wrong implication.
00:04:32.000 --> 00:04:34.000
Yeah, I try.
00:04:37.000 --> 00:04:38.000
No.
00:04:39.000 --> 00:04:42.000
But I love it every time I see it.
00:04:42.000 --> 00:04:46.000
I have done some stuff.
00:04:46.000 --> 00:04:59.000
Um, one thing very important to the show is I've been working on automation to make posting the announcements, um.
00:04:59.000 --> 00:05:06.000
much more, uh, friction-free, and I know everybody out there is saying.
00:05:06.000 --> 00:05:18.000
Jesus, Wolf. Um… Posting one paragraph on Mastodon every two weeks, what friction is there?
00:05:17.000 --> 00:05:20.000
Now you sound like me.
00:05:18.000 --> 00:05:19.000
Umm.
00:05:21.000 --> 00:05:22.000
Yeah.
00:05:21.000 --> 00:05:24.000
I think that's pretty much what I said.
00:05:24.000 --> 00:05:32.000
Um, but to me, there's friction, and it gets in the way, and you've seen the result that I don't do the posts.
00:05:32.000 --> 00:05:40.000
I'm not done with the automation, but I'll tell you all about it when I am done, and I will post the code.
00:05:40.000 --> 00:05:43.000
So there's that.
00:05:43.000 --> 00:05:47.000
I took this whole week off.
00:05:47.000 --> 00:05:53.000
Because I am preparing for my very biggest.
00:05:53.000 --> 00:06:00.000
um, match, ever. Uh, USPSA Level 2 match, the Michigan Sectionals.
00:06:00.000 --> 00:06:02.000
And.
00:06:02.000 --> 00:06:07.000
I thought I was gonna practice every day, all day long, and boy, I'm not doing that.
00:06:07.000 --> 00:06:32.000
Uh, I got my vaccines yesterday, I got the mRNA for the flu, and I got my COVID booster, and I slept today until 4pm! That doesn't sound right! So I don't know about that. Um… But it's been pretty good, I think.
00:06:31.000 --> 00:06:39.000
Yeah, yeah, you are definitely, you are definitely not an anti-vaxxer, and neither am I.
00:06:32.000 --> 00:06:34.000
So a gentle.
00:06:39.000 --> 00:06:48.000
Uh, yeah, um, I like to do this thing, we have a special word for it, and that word is science. Um…
00:06:47.000 --> 00:06:48.000
Yes.
00:06:49.000 --> 00:06:53.000
And a lot of people don't like science.
00:06:53.000 --> 00:06:57.000
Because when you use science.
00:06:57.000 --> 00:07:09.000
The answers are changing. As you learn more things, you refine what you think the proper course of action is.
00:07:09.000 --> 00:07:16.000
And there's a ton of people who are like, well, the answer changes, that means it's completely bogus, science is stupid.
00:07:16.000 --> 00:07:17.000
Umm.
00:07:19.000 --> 00:07:24.000
I have opinions, I think you know what they are.
00:07:26.000 --> 00:07:38.000
I think the right thing for us to do right now is to get right into the meat of the, uh, episode, and Jim has a topic.
00:07:38.000 --> 00:07:41.000
Jim, why don't you go ahead?
00:07:41.000 --> 00:07:54.000
Yeah, so, um, some of you know this, uh, and, and I'm sure some don't, but there's a problem, uh, lurking in our future, uh, and that's the Unix year 2038 problem.
00:07:55.000 --> 00:08:05.000
UNIX has historically stored timestamps in a 32-bit signed integer.
00:08:05.000 --> 00:08:16.000
00 UTC.
00:08:16.000 --> 00:08:35.000
Um… we're gonna… we're gonna overflow that integer on, um… uh… let's see, what's the date? I have it here. January 19th, 2038. So we've got 8 years, right? 7 and a half years. We're gonna… we're gonna overflow that integer.
00:08:35.000 --> 00:08:41.000
What's going to happen when that happens?
00:08:41.000 --> 00:08:59.000
Lots of things might happen. Some things people are worried about maybe won't happen. But what happens with a signed integer is when you keep incrementing, in this case, a signed 32-bit integer holds a number that is 2,147,483,647.
00:08:59.000 --> 00:09:02.000
That's the number of seconds.
00:09:02.000 --> 00:09:04.000
When you add one to that.
00:09:04.000 --> 00:09:10.000
It does not go to 2,147,483,648.
00:09:10.000 --> 00:09:18.000
Uh, no. It flips negative, because the integer overflows, the sign bit gets, uh, set.
00:09:19.000 --> 00:09:29.000
Uh, and then all the other bits go back to zero. But that number is negative $2,147,483.
00:09:29.000 --> 00:09:36.000
2,147,483,648.
00:09:36.000 --> 00:09:40.000
And then it starts counting down towards zero.
00:09:40.000 --> 00:09:41.000
Umm.
00:09:42.000 --> 00:10:02.000
It's… it's anybody's guess what's gonna happen at that point. Um… think of things, uh… you know, as a software developer, I use Makefiles. Make depends on file stamps on files, right? When you, uh… when you have an object file and a… and a… source file, and the source file is newer than the object file.
00:10:03.000 --> 00:10:10.000
uh, the, the, the code's gonna get compiled. Make will, will see that the source code has changed. Well.
00:10:10.000 --> 00:10:11.000
If, uh.
00:10:11.000 --> 00:10:20.000
Uh, the timestamp rolls over, or the current time rolls over, and all of a sudden, it thinks we're back in 1901?
00:10:20.000 --> 00:10:32.000
And you touch a file. Is that file newer than the object file that was last built in 2037?
00:10:32.000 --> 00:10:48.000
No, so your make is gonna fail. Now, that's not a huge problem, but there's all kinds of other problems that relate to timestamps and things happening or not happening, and age… calculations and stuff. So… It's a real problem.
00:10:48.000 --> 00:11:02.000
And this is, this is kind of a class of problems, and we're, we're gonna cover a few of them, a few more of them here. Um, but, uh, uh, there have been steps, uh, made to, to solve this problem.
00:11:02.000 --> 00:11:21.000
Um… It's, uh, on a, this is typically on a 32 bit system, a system that stores its time in a 32 bit timestamp, a 32 bit CPU, or even a 16 bit CPU that uses a long integer, uh, can have this problem. 64 bit systems.
00:11:21.000 --> 00:11:32.000
generally don't have this problem, uh, because an integer in 60… in a 64-bit CPU is 64 bits long. That's a much, much, much longer.
00:11:32.000 --> 00:11:48.000
Uh, uh, time. That's, that's billions of years, uh, worth of seconds. So that problem doesn't exist. Uh, but there's an awful lot of machines out there, uh, systems out there, uh, embedded systems out there that are running 32-bit CPUs.
00:11:48.000 --> 00:11:59.000
that, boy, they could run into this problem. But anyway, a lot of work has been done. As I mentioned, the 64-bit systems really don't have this problem.
00:12:00.000 --> 00:12:08.000
Linux, the Linux kernel for 32-bit CPUs, they took care of that.
00:12:08.000 --> 00:12:25.000
Uh, let's see, how's it being fixed? Uh, they took care of that in, uh, Linux kernel 5.6, which was, uh, released in 2020. So that kernel's been out for about 6 years. So even if you have a 32-bit system, as long as you've got a newer version of Linux, uh, you should be okay.
00:12:25.000 --> 00:12:34.000
Um, but also, and that's just the time T, uh, data structure, uh, field that, that holds the time. Um.
00:12:34.000 --> 00:12:42.000
if your software hasn't been compiled to take advantage of that, you could still have a problem.
00:12:42.000 --> 00:12:52.000
This problem… feels an awful lot like a problem we all ran through, uh, any of us who've been computing for a while, and that's the Y2K.
00:12:52.000 --> 00:13:07.000
problem. Remember the year 2000 bug? Wolf, did you have to worry about that much? See, I was working with medical information. I was working with financial transactions, patient data. The year was really important to us.
00:12:58.000 --> 00:13:00.000
I did.
00:13:07.000 --> 00:13:21.000
Uh, so… so we had to deal with this. Uh, the year 2K… the Y2K problem is really the same kind of thing. The field that was set up to store the year wasn't big enough to hold.
00:13:21.000 --> 00:13:30.000
a big year. And that was when the calendar flipped from 1999 to 2000.
00:13:30.000 --> 00:13:37.000
A lot of old software was written to store the year as a two-digit number.
00:13:37.000 --> 00:13:47.000
year. So when it went from 99 to 2000, when it went from 1999 to 2000, it really went from 99 to zero.
00:13:48.000 --> 00:14:04.000
So all of a sudden the year is zero instead of 99. So now age calculations were all off. They predicted all kinds of problems. Fortunately, engineers saw this for a long, long time coming. You know, they sort of knew it was going to happen.
00:14:04.000 --> 00:14:19.000
So an awful lot of work went into it. A lot of money was spent on this. And in the end, the problems were really minor. I don't know of any significant issues that happened when we flipped to the year 2000.
00:14:20.000 --> 00:14:38.000
Um… And some people will say, well, that was an awful lot of hype, an awful lot of work for nothing. And others will say, well, we didn't have major problems because we put all that work into making sure we didn't have that problem. I side with the latter, where I think, yeah, the work was done.
00:14:38.000 --> 00:14:40.000
Uh, in my case.
00:14:40.000 --> 00:14:56.000
I went to work for a company in 1985. And at that point, the software, it was all written in COBOL, and it had two digit years. So patient dates of birth, dates of service, financial transaction dates, they were all a two digit year.
00:14:56.000 --> 00:15:11.000
I was in a good place because, uh, it was 85. You know, we had 15 years to deal with the problem. And in around 92, we started redesigning the system. Complete, complete rewrite. This was all written in COBOL.
00:15:11.000 --> 00:15:16.000
Uh, complete, uh, rewrite, uh, so we could deal with that.
00:15:16.000 --> 00:15:22.000
Um… But, but let's talk for a minute about, about why.
00:15:22.000 --> 00:15:25.000
Dates were stored in a two-digit year.
00:15:25.000 --> 00:15:45.000
Uh, you gotta think about, uh, storage, uh, back in the day, back in the 60s and 70s, and even into the 80s. Storage was expensive. Um… and rather limited in size. Uh, you go back to the 60s and 70s, and storage, many times, was punch cards. Did you ever get to work with punch cards?
00:15:45.000 --> 00:15:47.000
Oh my god, yes.
00:15:47.000 --> 00:15:53.000
I'm dating myself, but I never dropped a deck.
00:15:53.000 --> 00:16:02.000
But I absolutely had to deliver decks to the computing center for them to run.
00:15:53.000 --> 00:15:55.000
No.
00:16:02.000 --> 00:16:04.000
And.
00:16:04.000 --> 00:16:05.000
Yah.
00:16:04.000 --> 00:16:19.000
Yeah, well, I had a little bit of experience with cards. I went to school at Michigan Tech, and at that time we were, you know, the Fortran class, we were learning, we were programming using punch cards. That was my only experience was in school.
00:16:19.000 --> 00:16:28.000
Uh, but boy, I remember, you go to the student bookstore and buy a box of punch cards. It was 2,000 punch cards.
00:16:28.000 --> 00:16:32.000
And… The big fear was you'd drop that box.
00:16:32.000 --> 00:16:39.000
And, you know, a program, you know, you know, classic Fortran program for a college student.
00:16:39.000 --> 00:16:54.000
Uh, you know, the early days of, uh, programming classes, our job was to create a payroll system that calculates payroll, you know, regular time at 40 hours, time and a half after 40 hours, so we had to write a program like that. And that program might have been 100 lines of code.
00:16:54.000 --> 00:17:03.000
But even that, 100 lines of punch cards, 100 cards, uh, because each punch card was 80 characters wide, um…
00:17:02.000 --> 00:17:05.000
Not counting JCL.
00:17:05.000 --> 00:17:21.000
Well, yeah, you had to, you had to add the JCL at the beginning. So those, each line of JCL was another punch card. So you might've had six or seven or eight lines or eight cards in front of your code and then a, a, a card or two at the end to, to indicate that's the end of the deck.
00:17:21.000 --> 00:17:37.000
Uh, but boy, occasionally you'd see somebody in the hall, and they tripped or something, and they dropped their cards, and uh, yeah, there's their, there's their 110 lines of code… scrambled.
00:17:37.000 --> 00:17:52.000
I, it was, uh, I, I, I shouldn't laugh 'cause it did happen to people. But anyway, you, you take that 80 character card, that's 80 characters of information you can store on it. And a card sometimes would represent, um, a student.
00:17:52.000 --> 00:18:05.000
You know, it'd have their name on there, and, uh, their date of birth, um, maybe, um… I don't know. Maybe their address? You know, you can't fit a whole lot into 80 characters.
00:18:05.000 --> 00:18:17.000
Uh, so what you would do is you would try to store things as tightly as you could. You wouldn't store 1980 as the year. You would store 80.
00:18:17.000 --> 00:18:24.000
Because that extra 2 bytes out of 80, that was important. If you had a couple of dates in the row, in that record.
00:18:24.000 --> 00:18:29.000
There was a lot of space, so people just would use two-digit years.
00:18:29.000 --> 00:18:30.000
Umm.
00:18:30.000 --> 00:18:35.000
Fast forward a little bit to hard disks. Um, you know.
00:18:35.000 --> 00:18:46.000
back in those days, well, in college, we didn't have hard disks, but when I first started working for real in software development, we had hard disks that were on the order of, like.
00:18:46.000 --> 00:19:02.000
five megabytes. We had floppy disks, you know, that were, that were, uh, 512 K. Um, you didn't have a lot of room. So, so every byte you could save was a lot. So again, uh, uh, the designers of software, they would just.
00:19:02.000 --> 00:19:04.000
skimp and go with the two digit years.
00:19:05.000 --> 00:19:09.000
The other problem with hard disks, even as the hard disks grew.
00:19:09.000 --> 00:19:26.000
Um… storage was… it's not like it is now. Everybody paid attention to storage, the layout of the data on the disk. So if you were creating a table with a bunch of records in it, you know.
00:19:26.000 --> 00:19:30.000
patient records, or transaction records, or something like that.
00:19:30.000 --> 00:19:31.000
Um.
00:19:31.000 --> 00:19:46.000
The way hard disks worked was you have a platter or a set of platters. You know, if you've ever taken a disc apart, you've seen them, you've probably seen pictures of them, right? You have this set of platters and each platter has these concentric rings and those are called tracks.
00:19:46.000 --> 00:20:02.000
and each… and there might be, I don't know, 63 tracks on a platter, and each track was a set of sectors. It was broken down into, like… I remember the old IBM hard drives were, like, 17 sectors for a track.
00:20:02.000 --> 00:20:14.000
And each sector was… There was varying sizes, but there were, like, 256 bytes, or 512 bytes, or, uh, newer drives were, like, 4,096 bytes.
00:20:14.000 --> 00:20:27.000
The systems were so crude back then that when you'd store data, you could only write a sector at a time, or read a sector at a time. It had to be a full sector. So if you wanted to write one byte, it was going to use up a full sector. It was going to write to that sector.
00:20:27.000 --> 00:20:37.000
Uh, so if you wanted to write a patient record, uh, and maybe it's, uh, you know, 240 bytes of information, it was gonna use a sector of 512 bytes.
00:20:37.000 --> 00:20:50.000
No matter what. If you wanted to write a second patient, it didn't get tacked onto the end of that sector, it used the next sector. So there was a lot of wasted space. So everybody was concerned with packing that data in as tightly as they could.
00:20:50.000 --> 00:21:12.000
Uh, boy, I'm glad we don't have to worry about that anymore, because that, uh… you really had to consider what you were doing. Sometimes you didn't get to store the data you wanted to store. If you had a 512-byte sector, and you had 513 bytes of data, you would use one full sector, and then you'd use the next sector just to store that one byte.
00:21:12.000 --> 00:21:21.000
See, you get where I'm coming from, right? Uh, storage was at a premium. So, decisions were made in the design of software.
00:21:15.000 --> 00:21:17.000
Uh-huh.
00:21:21.000 --> 00:21:29.000
to cram as much as you could into as small a space as you could. So two digit years were chosen.
00:21:29.000 --> 00:21:56.000
And that's what led to the Y2K bug. The year 2038 thing is a little bit different. That was a real hardware limitation. Well… They had chosen to store the timestamp in a 32-bit integer. They didn't have 64-bit integers at the time, and they didn't want to go with an unsigned integer, because in 1970, every number going forward was a date in the future.
00:21:56.000 --> 00:22:01.000
Every number that was negative was a date and time in the past.
00:22:01.000 --> 00:22:12.000
You know, some people said, well, you know, can't we just change the time t data structure to make it an unsigned integer? Well, then you lose all the dates before 1970.
00:22:12.000 --> 00:22:15.000
So… It was an issue.
00:22:15.000 --> 00:22:17.000
Umm.
00:22:18.000 --> 00:22:34.000
I don't know. It had caused a lot of trouble, and a lot of money was spent back on Y2K, and now I think we're going to start hearing about this Y year 2038 thing.
00:22:35.000 --> 00:22:50.000
Fortunately, like I said, 64-bit systems really don't have the problem. 32-bit systems that are recompiled with a 64-bit time field don't have the problem. But all of those embedded systems out there.
00:22:50.000 --> 00:22:58.000
There's things, you know, things you don't even think about. Traffic lights. Um, I don't know, you got some idea of, uh, embedded systems, Wol.
00:22:58.000 --> 00:22:59.000
And you?
00:23:01.000 --> 00:23:03.000
You want me to name some?
00:23:02.000 --> 00:23:04.000
Uh, yeah!
00:23:05.000 --> 00:23:16.000
Oh my god, so many, everything in your car, everything in, you said, traffic lights, everything in.
00:23:16.000 --> 00:23:21.000
Utility management like water.
00:23:19.000 --> 00:23:31.000
Yeah, that's the big thing. Those things that have been out there for a long, long time, monitoring water flow, oil pressure, all those things out in the field that nobody can even get to.
00:23:31.000 --> 00:23:34.000
Those systems could fail.
00:23:34.000 --> 00:23:36.000
Yeah, okay.
00:23:36.000 --> 00:23:42.000
And, um, I have a take on this.
00:23:42.000 --> 00:23:50.000
Uh, you said… Thankfully, we don't have to care about this anymore. I would say…
00:23:49.000 --> 00:23:51.000
Yeah.
00:23:50.000 --> 00:23:55.000
Um, we don't HAVE to care about this anymore.
00:23:55.000 --> 00:24:07.000
But we should. There are things about storage and locality and size and packing that remain important.
00:24:07.000 --> 00:24:10.000
But they are no longer urgent.
00:24:10.000 --> 00:24:16.000
Uh, but we still need to consider them, and because we don't HAVE.
00:24:10.000 --> 00:24:12.000
Yah.
00:24:16.000 --> 00:24:17.000
You have to.
00:24:17.000 --> 00:24:25.000
People want to skip that, and I think skipping it is wrong.
00:24:23.000 --> 00:24:38.000
That's where… that's where bloat comes from, right? That's… that's the cause of bloat. People not paying attention to how they're… how they're storing their data, whether it's on a storage medium or in RAM, uh, in the system.
00:24:39.000 --> 00:24:53.000
Um, yeah, it's, uh, it's interesting. Okay, here's another, uh, similar situation to the year problem, uh, whether it's the year 2038 problem or the Y2K problem. Did you ever hear something called the DJ10K problem?
00:24:54.000 --> 00:24:58.000
DJ 10K, I have not?
00:24:55.000 --> 00:24:58.000
DJ 10K.
00:24:58.000 --> 00:24:59.000
When is that?
00:24:58.000 --> 00:25:16.000
Dow Jones Industrial Average. Uh, back in the 90s, uh, the, the Dow Jones, uh, Industrial Average was in the thousands, maybe, uh, 6,000, 7,000. It's a, it's a scoring, or, uh, uh, uh, of, uh.
00:25:16.000 --> 00:25:18.000
It's…
00:25:16.000 --> 00:25:22.000
Sure, I see it all the time. It is constantly listed.
00:25:18.000 --> 00:25:22.000
It's an indicator, yeah, you hear it in the news all the time, it's.
00:25:22.000 --> 00:25:35.000
yeah, it's an indicator of how the market is doing, right? Well, back then, it was around 6,000, 7,000, uh, and people started to worry, what's gonna happen when it hits 10,000?
00:25:35.000 --> 00:25:46.000
A lot of software out there was written to only allow 4 digits for the Dow Jones Industrial Average. So, uh, trading systems?
00:25:43.000 --> 00:25:53.000
I can't wait for you to get to the part where I get to say, oh, nobody is ever going to need more than 640K of RAM.
00:25:50.000 --> 00:25:53.000
Yeah. Ha ha.
00:25:53.000 --> 00:26:08.000
Right, right. Well, nobody's ever gonna… nobody's ever gonna worry about the Dow Jones average going above 10,000. Nobody's ever gonna worry about a year hitting 2038, you know, back when it's 1970.
00:26:08.000 --> 00:26:14.000
That's 68 years in the future. Nobody's possibly going to be using this software in 2038, are they?
00:26:14.000 --> 00:26:24.000
You know, in the 60s, they were writing healthcare systems, and nobody was going to worry about them still using that software in the year 2000.
00:26:14.000 --> 00:26:40.000
Right. Yeah, this… This just points out to me a thing that I have said many, many times on many, many topics, which is there's two different kinds of decisions you will make as a software engineer.
00:26:41.000 --> 00:26:46.000
One is, you'll make a decision based on measurement.
00:26:46.000 --> 00:26:55.000
and everything you know scientifically. Is this going to need x?
00:26:55.000 --> 00:27:05.000
Um, is this the slow part? Um, is this a thing that would make more sense if I put an index on it? Blah blah blah blah.
00:27:05.000 --> 00:27:12.000
And the other decision is when you make a decision based on what you feel.
00:27:13.000 --> 00:27:16.000
Now, I don't want to discount feelings.
00:27:16.000 --> 00:27:22.000
But the fact is, the first one is better.
00:27:22.000 --> 00:27:37.000
Everything you feel is suspect. When you feel that no one is going to need more than $640K, when you feel that your software is not going to be used.
00:27:25.000 --> 00:27:27.000
Hmmm.
00:27:37.000 --> 00:27:43.000
for 12 more years when you feel that a two-digit year is enough.
00:27:43.000 --> 00:27:44.000
Umm.
00:27:45.000 --> 00:27:57.000
Maybe that decision is right, maybe it's wrong, but your reason for making that decision absolutely is wrong.
00:27:57.000 --> 00:28:06.000
Um, you should have measured. You should have figured out what the reality was. Um, and maybe you would have come to the same conclusion.
00:28:06.000 --> 00:28:08.000
But maybe you wouldn't.
00:28:08.000 --> 00:28:10.000
Yeah, uh…
00:28:08.000 --> 00:28:10.000
That's my feeling.
00:28:10.000 --> 00:28:18.000
Yeah, but if we go back to the 60s, you know, when people were… I mean, computing was in its… just its infancy.
00:28:18.000 --> 00:28:30.000
Right? And storage, like I said, was at a premium. I don't really mean to be defending what they did, but they kind of did what they had to do, right?
00:28:30.000 --> 00:28:32.000
Uh, and, and, you know.
00:28:33.000 --> 00:28:44.000
The software industry was so young, they didn't think it was at all possible that these programs they were writing would still be there.
00:28:41.000 --> 00:28:42.000
I hear you.
00:28:42.000 --> 00:28:49.000
I hear you. But there are things in this world that are hard.
00:28:44.000 --> 00:28:45.000
But, and here we are.
00:28:49.000 --> 00:28:54.000
But you still have to do them. For instance, um…
00:28:51.000 --> 00:28:53.000
Sure, sure.
00:28:54.000 --> 00:29:00.000
Two things that I value are integrity and courage.
00:29:00.000 --> 00:29:01.000
Mmhm.
00:29:00.000 --> 00:29:08.000
And it turns out that to do the thing that satisfies integrity.
00:29:08.000 --> 00:29:14.000
sometimes requires courage because you have to do a thing.
00:29:11.000 --> 00:29:13.000
Sure.
00:29:14.000 --> 00:29:21.000
That was harder, that you maybe had to defend to your bosses.
00:29:21.000 --> 00:29:32.000
But it was absolutely the right thing to do, even though it felt to them like it wasn't the right thing to do.
00:29:32.000 --> 00:29:33.000
Right now.
00:29:33.000 --> 00:29:34.000
Sure.
00:29:33.000 --> 00:29:34.000
Umm.
00:29:34.000 --> 00:29:38.000
And it was harder, and that's just the way it is.
00:29:38.000 --> 00:29:50.000
Yeah, well, you know, some… yeah, sometimes you're just trying to solve today's problem without thinking about, uh, the problem… what problem's gonna create in 20 years. It's… it's…
00:29:48.000 --> 00:29:53.000
I agree with that, but that's a junior programmer.
00:29:50.000 --> 00:29:51.000
It's… yeah.
00:29:53.000 --> 00:30:02.000
Sure, sure. And, you know, in the 60s, I don't think there was much other than junior programmers.
00:30:02.000 --> 00:30:07.000
You know, there weren't a lot of senior-level software developers.
00:30:03.000 --> 00:30:05.000
I agree with that.
00:30:07.000 --> 00:30:24.000
Right? Anyway, so we'll get back to this DJ 10K problem real quick. Imagine the Dow Jones Industrial Average is ticking upward, upward, upward.
00:30:07.000 --> 00:30:09.000
Yeah, I agree with that.
00:30:24.000 --> 00:30:38.000
And it gets to the 9,000s. Uh, 9,900. It's getting really close. Um… If the market is depending on that number to decide what to do.
00:30:38.000 --> 00:30:46.000
And all of a sudden, that number is more than 5 digits, and it can't be represented in a 4-digit field.
00:30:46.000 --> 00:31:03.000
They didn't know what was going to happen. The possibilities were the leading digit would get truncated. So now 10,102, let's say, is the number. If it gets truncated, it could become.
00:31:03.000 --> 00:31:07.000
102, because that first digit got chopped off.
00:31:07.000 --> 00:31:23.000
Well, that's like the market crashing, right? If the industrial average goes from 9,999 to 100, major sell-offs could happen. That could be catastrophic.
00:31:10.000 --> 00:31:12.000
It is.
00:31:23.000 --> 00:31:30.000
Especially because of all this automated software that buys and sells on its own.
00:31:23.000 --> 00:31:45.000
Um… Yeah, and, and, uh, well, the number did cross that threshold on, um, March 29th, 1999. And fortunately, there were no real big problems, because the engineers had.
00:31:45.000 --> 00:31:50.000
Had gone through and did the analysis, and they updated software to make sure it wouldn't happen.
00:31:50.000 --> 00:32:08.000
But, uh, you know, with automated trading, I don't know how much automated trading they were doing in 99, uh, but now it's a lot of automated training. But imagine, uh, the automated training all of a sudden sees that the number dropped down to 102.
00:32:08.000 --> 00:32:23.000
and it starts this automatic sell-off, well, now there are triggers in place that will stop. It'll just freeze the market if that happens for… I don't know how long it has to happen for. I think it's in a matter of seconds.
00:32:23.000 --> 00:32:25.000
The markets will actually halt.
00:32:25.000 --> 00:32:54.000
And that's a problem! You know, now you have unstability in the market. Big problems. Fortunately, they were able to, uh… not experience any issues, because they did work on it, but a lot of money, I'm sure, was spent to deal with that, and a lot of… it created its own instability, just knowing that that was going to happen. I remember this in 99, this was happening, because everybody was focused on Y2K.
00:32:54.000 --> 00:33:01.000
And then it's like, oh God, now here's another thing. But fortunately, neither one of those turned into much of a problem.
00:33:02.000 --> 00:33:03.000
Up.
00:33:03.000 --> 00:33:19.000
So I did mention how this problem is being fixed in Unix systems 64 bit systems don't have the problem. Newer Linux kernels don't have the problem. As long as your program is recompiled with the larger time, the larger time T.
00:33:19.000 --> 00:33:35.000
Uh, data type, uh, what is it a structure? I, I, I, I don't really know how it's stored. Um, I, I think it's just a, just a, a, a, an integer that, like I said, it is the number of seconds, uh, file systems.
00:33:35.000 --> 00:33:50.000
are a problem. Fortunately, the ext4 file system, which is probably the most popular file system on Linux, it handles it. Although there's a choice you can make with the ext4, whether you want to go with large inodes or small inodes.
00:33:50.000 --> 00:34:07.000
And if you go with the small iNodes, you're stuck with 32-bit timestamps. So that's a problem. But I think the default is the 64-bit timestamps for that. So that's good. But then there's other file systems, BTRFS, XFS. I can't even think of the others right now, but they all handle it just fine.
00:34:07.000 --> 00:34:14.000
The older EXT3 file systems? No, it's a 32-bit timestamp. Uh, those will be problems.
00:34:14.000 --> 00:34:19.000
I usually go with ButterFS.
00:34:17.000 --> 00:34:19.000
Do you know?
00:34:19.000 --> 00:34:24.000
I do, um, because supposedly it's better.
00:34:24.000 --> 00:34:33.000
Yeah, I hear it's faster. I think it's only recently that you could install the operating system on a ButterFS.
00:34:33.000 --> 00:34:35.000
Was that you clicking?
00:34:35.000 --> 00:34:37.000
Accidentally, yes.
00:34:36.000 --> 00:34:38.000
Okay.
00:34:38.000 --> 00:34:41.000
We'll just… we'll tell the editor to leave that in.
00:34:44.000 --> 00:34:56.000
Um, uh, it's only recently that a ButterFS, uh, you can use a ButterFS file system as the root file system. Uh, I think, uh, Ubuntu 2024 added that?
00:34:56.000 --> 00:35:01.000
Um, before that, your only choice was, like, an EXT4.
00:34:57.000 --> 00:35:05.000
And for everybody listening who doesn't know what I mean when I say ButterFS, I mean BTRFS.
00:35:05.000 --> 00:35:06.000
Yeah, yeah.
00:35:06.000 --> 00:35:17.000
Uh, BTRFS is pretty interesting. I think it supports snapshots. XFS supports snapshots. There's all kinds of neat features with these file systems. Didn't we do an episode on file systems? I think we did.
00:35:16.000 --> 00:35:18.000
I think we did.
00:35:17.000 --> 00:35:36.000
It was… yeah, that was like a year ago. Um, anyway… Let's talk about databases for a minute. You know, I'm a big fan of Postgres. I use Postgres all the time, for everything. And we store timestamps in the Postgres timestamp TZ data type.
00:35:36.000 --> 00:36:07.000
Well, that's an 8-bit, uh, an 8-byte, 64-bit number. That will store a timestamp all the way out to the year 294,276. So… You know, it might have been shortsighted to think the software would still be running after 2038. I think it's, I think it's pretty likely you're going to be okay by the year 294,276. So that's how I store all my timestamps and I store my dates in a date field.
00:36:07.000 --> 00:36:21.000
So I don't have to worry about two-digit year, four-digit year, whatever. Um, that's Postgres. MySQL, um, they've… They've got a few issues. Their timestamp is a 32-bit timestamp.
00:36:21.000 --> 00:36:36.000
It's a signed number, so it suffers the same problem that, you know, the original Unix 32-bit timestamp has. MySQL does have a date/time type that stores the number as a 5-byte byte.
00:36:36.000 --> 00:36:45.000
packed binary, uh, and that'll take you through, all the way through the year 9999. So that's, that's pretty good.
00:36:45.000 --> 00:36:51.000
Um, I… I was really interested to learn this. SQLite doesn't have a date or a time.
00:36:51.000 --> 00:36:52.000
Type.
00:36:52.000 --> 00:37:10.000
uh, you make a choice. You can store your, your timestamps as a, uh, ISO 8601 string, which is, you've probably seen this before, the YYYY-MM-DD, uh, hours, minutes, seconds, and then, um, up to thousands of a second, milliseconds.
00:37:10.000 --> 00:37:15.000
Um, you store it as a string in SQLite. Um…
00:37:14.000 --> 00:37:18.000
I wonder what DuckDB uses.
00:37:19.000 --> 00:37:20.000
I don't know.
00:37:20.000 --> 00:37:23.000
I don't know. I'll bet there's an easy way.
00:37:23.000 --> 00:37:35.000
Uh, I did look, uh, uh, SQL Server, uh, they've got fancy types for that, uh, type. Does DuckDB use for timestamps?
00:37:32.000 --> 00:37:39.000
It's funny to me that you group SQL Server with databases.
00:37:40.000 --> 00:38:01.000
Well, it is a database. I mean, so is Oracle. They're just not databases that I use. I have used SQL Server. Um… Uh, DuckDB stores… they've got a timestamp and a timestamp TZ, uh, field that stores it as the number of microseconds since, uh, 1970-01-01, uh, the Unix epoch.
00:38:00.000 --> 00:38:02.000
And how big is it?
00:38:02.000 --> 00:38:07.000
Um, you know what? It's not telling me.
00:38:07.000 --> 00:38:09.000
Uh, let me ask you.
00:38:08.000 --> 00:38:11.000
I don't like it when it doesn't tell you.
00:38:11.000 --> 00:38:16.000
How many bits, uh… this is 64-bit.
00:38:16.000 --> 00:38:25.000
So that's good. So, uh… Uh, that'll… that'll get you out pretty far. Way, way farther than a 32-bit. So that's… that's pretty nice.
00:38:25.000 --> 00:38:26.000
Umm.
00:38:27.000 --> 00:38:36.000
So, yeah, uh, if you're gonna store a date in a system, uh, use a database, and use their date.
00:38:36.000 --> 00:38:53.000
field, if you can. If you've got a timestamp, store it in a timestamp field, but if you're not sure what the storage format is, check it out, just to make sure it'll support what you're trying to do. I gotta think that databases in this day and age, you're gonna be.
00:38:53.000 --> 00:38:55.000
pretty good. You know.
00:38:55.000 --> 00:39:00.000
Back when I was doing the software in when I first started in 1985.
00:39:00.000 --> 00:39:15.000
Um, I said before, we were using COBOL. COBOL didn't have any kind of… COBOL didn't have types. You had numbers and text, and that was it. Uh, you had no date type or timestamp type.
00:39:15.000 --> 00:39:26.000
Um, so that's, that's, uh, that's, you know, that's much better now, when we actually have, uh, programming languages that actually have date time, uh, uh, types, uh, and timestamp types.
00:39:27.000 --> 00:39:43.000
Um, but, uh, this, this whole conversation leads us to a whole bunch of overflow issues like this, because that's what it was, an overflow issue, right? So, I, I've got a list of them, I'm just gonna run down them, uh, pretty quickly. Uh, you know, uh, GPS.
00:39:43.000 --> 00:39:59.000
has an issue. I actually remember hearing about this, uh, this happened in 2019. Uh, GPS systems, you know, those satellites up there, uh, you can't go up there and change them very easily, uh, but they use a 10-bit.
00:39:59.000 --> 00:40:01.000
Weak counter.
00:40:01.000 --> 00:40:12.000
That's week, like, like, you know, this week, next week, counter. Um, uh… 10-bit, so it runs out roughly every 20 years.
00:40:12.000 --> 00:40:29.000
And when that happens, I think a reset has to happen, and your GPS devices may lock up, you might have to reboot them, and then they start working again. Well, it happened in 1999.
00:40:29.000 --> 00:40:45.000
It happened in 2019. It's going to happen again in 2038. I don't know if it coincides with… I think it's completely independent of the year 2038 problem. And it's going to happen in 2058. They are changing it to a 13-bit weak counter.
00:40:45.000 --> 00:40:48.000
But, of course, that involves software updates.
00:40:45.000 --> 00:40:47.000
Oh, good.
00:40:49.000 --> 00:40:57.000
Well, 3 more bits, that's a lot more, right? That's pretty good. Um, isn't that 3 orders of magnitude larger?
00:40:57.000 --> 00:40:59.000
It is.
00:40:57.000 --> 00:40:59.000
So.
00:40:59.000 --> 00:41:01.000
Yeah, so instead of, uh…
00:40:59.000 --> 00:41:05.000
Three orders of magnitude, where an order of magnitude is base two.
00:41:03.000 --> 00:41:08.000
is a power of two. Yeah, yeah. Anyway, it's much better.
00:41:08.000 --> 00:41:14.000
Um, did you know NTP, the Network Time Protocol, uh, has a rollover issue?
00:41:14.000 --> 00:41:26.000
Uh, they use… they do! They use a 32-bit unsigned field. Uh, in this case, it's unsigned. Uh, it holds the number of seconds since 1900.
00:41:15.000 --> 00:41:17.000
I did not.
00:41:26.000 --> 00:41:36.000
Uh, it rolls over roughly every 136 years. So guess what? Not only do we have the 2038 problem, there's also gonna be a 2036 problem.
00:41:36.000 --> 00:41:44.000
And all of our systems, all of my systems use NTP to keep the clock set correctly.
00:41:44.000 --> 00:42:01.000
Uh, your systems probably do it too. If you don't even set it up, I'm sure it's there, right? And all these network devices, uh, routers and stuff, they all use NTP to keep the time going. Well, February 7th, 2036, look out, because they're all gonna roll over, and who knows what's gonna happen.
00:42:01.000 --> 00:42:20.000
I mentioned many times, and I even mentioned it earlier, I'm a big Postgres fan. Well, Postgres has an overflow issue. They've had it for years and years and years. Fortunately, they deal with it pretty well now. But they have a transaction ID, it's called the XID. And every time you start a transaction, this counter gets incremented.
00:42:20.000 --> 00:42:25.000
Uh, and any records that you write get stamped with that counter.
00:42:25.000 --> 00:42:38.000
as part of that transaction, so that they're multi-version… what is MVC-C? Multi-version Currency Control. Um… Uh, make sure that you get the right snapshots.
00:42:34.000 --> 00:42:51.000
This is weird to me. Like, why do transaction IDs, which don't get stored en masse, uh, have to be integers? Why can't they be orderable, um… U U I D's.
00:42:50.000 --> 00:42:56.000
Well… They can. That's just not what… in the 90s, that's not what they chose.
00:42:56.000 --> 00:43:12.000
That's, that's the problem. So, and there's a lot of Postgres databases out there. Uh, Postgres actually, uh, the way they solve the problem is, uh, with Postgres, if you, um, well, first of all.
00:43:12.000 --> 00:43:27.000
if you run out of these transaction IDs, these XIDs, it's a number that's, you know, 2 to the 31st minus 1, is that how it works? So it's 4.3 billion, roughly. When you hit the end, it wraps around.
00:43:27.000 --> 00:43:31.000
And any data that you have that's stamped with that…
00:43:28.000 --> 00:43:35.000
It's very surprising to me that a transaction ID would be signed.
00:43:31.000 --> 00:43:33.000
Yeah, thank you.
00:43:36.000 --> 00:43:37.000
It's unsigned.
00:43:38.000 --> 00:43:39.000
Did I say signed?
00:43:39.000 --> 00:43:41.000
Yeah, it's unsigned.
00:43:41.000 --> 00:43:42.000
Okay.
00:43:41.000 --> 00:44:22.000
Um, 4 billion. If it were signed, it'd be 2.1 billion. Uh, anyway, it's, uh, uh… What happens is, when it wraps around, any rows that had, uh… some value… it's weird, I tried to understand it. The, uh… the transaction ID, it gets split into… it's this rolling thing of 2 billion, so everything that's… greater than 2 billion is in one half, and everything that's less than 2 billion is in the other half. It's complex. Anyway, those rows, they're sitting there on disk, but your select won't find them.
00:44:23.000 --> 00:44:33.000
So that's a real problem. So Postgres detects that, and when it sees a transaction ID rollover, uh, or wraparound, uh, it'll halt the system.
00:44:33.000 --> 00:44:37.000
for fear of losing data. It will just halt the system.
00:44:36.000 --> 00:44:37.000
Mmhm.
00:44:38.000 --> 00:44:58.000
So, they solved this in Postgres 8, which was… 2005, so this problem was really fixed 20 years ago. They solve it by running vacuum. In fact, the problem has always been solvable by running vacuum. In Postgres, you have this.
00:44:58.000 --> 00:45:14.000
verb you run, it's called vacuum, that will sweep through the tables and clean things up. And it will clear out the transaction IDs for really old records that are absolutely no longer part of any transaction. Those transactions had committed long ago.
00:45:14.000 --> 00:45:16.000
So it clears the transaction IDs.
00:45:14.000 --> 00:45:23.000
And when I'm adding millions of records to specific tables, I absolutely do a vacuum on that table.
00:45:19.000 --> 00:45:20.000
Yes.
00:45:23.000 --> 00:45:25.000
At the end.
00:45:23.000 --> 00:45:33.000
You absolutely want to do that. That's for a different reason. The reason why you do that vacuum is to update your statistics, so that your query planner can do the right thing.
00:45:26.000 --> 00:45:28.000
Yes.
00:45:30.000 --> 00:45:31.000
Right.
00:45:33.000 --> 00:45:42.000
But, uh, and you're also… you're doing one transaction, I hope, when you do that, or many transactions, but not a transaction for every row?
00:45:42.000 --> 00:45:47.000
Cause that would be, that would be not optimal.
00:45:42.000 --> 00:45:44.000
Correct.
00:45:44.000 --> 00:45:57.000
Generally, if I'm doing millions or billions of changes to a single table, I am using multiprocessing, um, but I will break it up into.
00:45:51.000 --> 00:45:52.000
Umm.
00:45:55.000 --> 00:45:56.000
Yep.
00:45:57.000 --> 00:46:06.000
you know, 160, uh, transactions, where a process handles each one.
00:45:58.000 --> 00:46:00.000
groups.
00:46:01.000 --> 00:46:02.000
Sure.
00:46:02.000 --> 00:46:04.000
Sure.
00:46:05.000 --> 00:46:32.000
Yeah, yeah, you would do that, but you wouldn't do each one without… you wouldn't do it at all without transactions, I believe. But either way, um… uh… what Postgres added in 2005 was autovacuum, because… because sysadmins would forget to run vacuum, and then they'd run into this wraparound issue. So they added autovacuum, and it's this daemon that runs.
00:46:32.000 --> 00:46:48.000
when the system is not terribly busy, and it'll vacuum the tables for you. And that pretty well takes care of the problem. They are working on increasing the size of that transaction ID for some future version. I think since they added auto-vacuum.
00:46:48.000 --> 00:46:52.000
20 years ago, it isn't as much of a big deal.
00:46:52.000 --> 00:46:53.000
Okay.
00:46:53.000 --> 00:47:04.000
Here's one that I think is really interesting and maybe a little scary. And I heard about this several months ago. The Boeing 787, that big airplane that they fly.
00:47:04.000 --> 00:47:07.000
Uh, it has a 51-day bug.
00:47:04.000 --> 00:47:06.000
Uh-huh.
00:47:08.000 --> 00:47:21.000
If the generator runs continuously for 51 days without a power cycle, an internal counter overflows, and all four generator control units go into a failsafe simultaneously.
00:47:21.000 --> 00:47:24.000
Oh, I'm never gonna fly again.
00:47:25.000 --> 00:47:34.000
I mean, it's easy, you just reboot the generators, or you cycle the power on the generator, but do you want to be doing that at…
00:47:29.000 --> 00:47:30.000
Uh-huh.
00:47:32.000 --> 00:47:37.000
Sounds like my infotainment system in the car.
00:47:35.000 --> 00:47:52.000
Do you want to be doing that at 36,000 feet? I don't know. That's an interesting one. NASA's deep space probe, sorry, deep impact space probe that they sent up, I don't even know where that thing was going.
00:47:38.000 --> 00:47:41.000
I do not.
00:47:52.000 --> 00:48:02.000
probe the, uh, the far reaches of the universe, I think. Um, it has a 32-bit, uh, counter that counts tenths of a second.
00:48:02.000 --> 00:48:07.000
Uh, it overflowed, and NASA lost contact with the probe entirely.
00:48:08.000 --> 00:48:13.000
The probe is no longer reachable because of this counter.
00:48:10.000 --> 00:48:19.000
That was totally predictable. They knew how long the trip was. Absolutely they knew.
00:48:13.000 --> 00:48:15.000
Yeah.
00:48:17.000 --> 00:48:18.000
Yeah.
00:48:19.000 --> 00:48:21.000
Yeah, yeah.
00:48:21.000 --> 00:48:38.000
I don't know if, uh, you know, you know, NASA designs these missions, and they put a lifetime on these missions. You know, like the Mars rovers, you know, they might say it's gonna be there for 18 months, and then everybody's surprised, because it's still going 10 years later. And this, uh, deep impact probe, it might have been.
00:48:38.000 --> 00:48:40.000
Way beyond its lifetime.
00:48:40.000 --> 00:48:50.000
When it, when it ran into that, uh, when it overflowed the counter, but, uh, it's kind of embarrassing when that's the problem, right?
00:48:49.000 --> 00:48:53.000
Yeah, I can only think of one reason.
00:48:53.000 --> 00:49:09.000
And that reason, I mean, you know, technically junior programmers, but that reason is the person who wrote the code didn't think like this is going to last longer than X what they thought was.
00:48:53.000 --> 00:48:54.000
Yep.
00:49:09.000 --> 00:49:14.000
I'm not going to be here in X.
00:49:12.000 --> 00:49:13.000
Yes.
00:49:13.000 --> 00:49:19.000
I'm sure. I'm sure that happens. It's not going to be my problem.
00:49:19.000 --> 00:49:21.000
Right.
00:49:21.000 --> 00:49:38.000
Um… you know, in UNIX and Linux, we've got PIDs, the process ID numbers, and we have file descriptors, and we even have port numbers, and they're all in fairly small fields, like a PID. I think Linux.
00:49:30.000 --> 00:49:32.000
Uh-huh.
00:49:38.000 --> 00:49:52.000
Uh, now, on a 64-bit system, it's not a big deal, but on a 32-bit system, the PIDs, I think, were limited to, like, up to, uh, 65535. That's a 16-bit number, right? I think a process ID could be no bigger.
00:49:49.000 --> 00:49:51.000
Right and.
00:49:52.000 --> 00:49:54.000
Then 65,535.
00:49:52.000 --> 00:50:01.000
Like a reasonable thing about that size is PIDs reset when you reboot.
00:50:00.000 --> 00:50:17.000
Yeah, they wrap, and when a process dies, it can become available, you know, for the next one. So, I mean, it's a limit, but is it a big one? I don't know. Port numbers, you know, port numbers were, um… I think the TCPIP spec.
00:50:17.000 --> 00:50:50.000
Uh, it says that the port number is a 16-bit number. Um… And when you have a system that can take millions of connections, well, guess what? You've got a problem. You know, and these systems these days, they can take… an awful lot of simultaneous connections. Um, and… but the port number is limited, so, uh… it can be a problem. Now, normally, incoming connections, they're all gonna connect to, like, port 443, or port 80, or, you know, whatever the port happens to be, so it's not a problem. It's… it…
00:50:49.000 --> 00:50:53.000
22 would be the one for me.
00:50:52.000 --> 00:51:04.000
22 is the one for you. Yeah, but if you've got a system with millions of connections coming in on port 22, that's an interesting one. 22 is SSH.
00:51:01.000 --> 00:51:03.000
Yes.
00:51:04.000 --> 00:51:08.000
Anybody doesn't know. Uh, so that's an issue, right?
00:51:08.000 --> 00:51:13.000
Um, here's one that you've probably never heard of, the Snowflake ID Overflow.
00:51:13.000 --> 00:51:16.000
Oh, I absolutely never heard of that one.
00:51:13.000 --> 00:51:28.000
And you might be thinking, what the heck is that? Well, it was created by Twitter. It's the ID number that they use for posts and all kinds of things. It's a 64-bit number.
00:51:28.000 --> 00:51:30.000
Um, and… It.
00:51:30.000 --> 00:51:45.000
it can wrap around. I don't know that Twitter has had this problem, but Discord followed Twitter, and they used this 64-bit snowflake ID, and they've had serious concerns about it.
00:51:45.000 --> 00:51:56.000
But here's where it gets interesting. 64-bit number's big. You're never gonna run out of whatever you can store in 64 bits. Well, I say never, but I shouldn't say that. Here's the problem.
00:51:56.000 --> 00:52:00.000
A lot of this software now is being implemented in JavaScript.
00:52:00.000 --> 00:52:08.000
Did you know that a JavaScript 60… an integer in JavaScript is not 64 bits?
00:52:08.000 --> 00:52:14.000
it's 53 bits, or I guess 54 bits. It can hold up to.
00:52:14.000 --> 00:52:18.000
2 to the 53rd minus 1. It's about 16 digits.
00:52:16.000 --> 00:52:28.000
I'm a JavaScript expert, and I, you know, I worked on the original JavaScript interpreter at Netscape, and I did not know that.
00:52:19.000 --> 00:52:21.000
Yes.
00:52:22.000 --> 00:52:24.000
Yeah.
00:52:25.000 --> 00:52:26.000
I know.
00:52:27.000 --> 00:52:29.000
You didn't know this.
00:52:29.000 --> 00:52:41.000
You didn't know that. I didn't know that. Um, it just really, really surprised me to find out that the largest integer that can hold is roughly 16 digits long.
00:52:41.000 --> 00:52:57.000
Um, so software that's expecting a 64-bit number, and really it's getting a 54-bit number, that could be a real problem. And that's, uh, what Discord is, is battling right now, because they're using the Snowflake ID, uh, which is supposed to be 64 bits, but JavaScript.
00:52:57.000 --> 00:53:01.000
It stops at 54 bits.
00:53:01.000 --> 00:53:19.000
So, as long as you're dealing with it, you know, JavaScript is kind of funny, because it's sort of typeless in a lot of ways, and you can store numbers as strings, and as long as you're storing this 64-bit number as a character string, you're okay. But if you try to store it as a real integer, and maybe do some arithmetic on it.
00:53:19.000 --> 00:53:24.000
Even something as simple as adding one to it, that could be a problem.
00:53:24.000 --> 00:53:32.000
Um, another one, we talked about this, uh, a year ago, um, IPv4 exhaustion.
00:53:32.000 --> 00:53:53.000
Uh, IPv4 address is, again, a 32-bit address, and it's not even like we really get, um, uh… what can you hold in a 32-bit number? Well… 4 billion, right? It's not like we get 4 billion addresses, because the way they've segmented that address space up, we get quite a bit less than 4 billion addresses.
00:53:53.000 --> 00:54:09.000
Uh, and as we talked about in, uh, episode 11, the, the solution to that is, uh, IPv6. There's some other solutions. They, they, they broke address space up into CIDR blocks, and they got private addresses and stuff, and that's how we got this far, you know, because everybody's still using IPv4.
00:54:09.000 --> 00:54:20.000
But, um, there's an awful lot of IPv6 out there. In fact, Google, I think I mentioned this a month or two ago, Google announced that more than half of their traffic is IPv6.
00:54:21.000 --> 00:54:30.000
Um, okay, finally, I've got one. Um, the AACS DRM specification using TLS certificates.
00:54:30.000 --> 00:54:36.000
Um, it's a 32-bit hard-coded expiration field.
00:54:36.000 --> 00:54:54.000
Uh, that at some point, and this is not every device, but some, at some point, it'll start rejecting valid discs when it hits that expiration date. So you may have a Blu-ray player, and you might have a Blu-ray disc, and it might be working just fine today.
00:54:54.000 --> 00:55:10.000
But tomorrow, it might just decide to stop because it decides that you're not licensed to play that video on that DVD. Basically, your smart device becomes e-waste on a schedule that nobody planned.
00:55:11.000 --> 00:55:24.000
Uh, so I… I do have a couple of more that aren't exactly overflow issues, these are date bug issues, and… and these I've heard about before as well. Uh, you know, the Excel and Lotus spreadsheet, they've got a date issue?
00:55:25.000 --> 00:55:31.000
Did you know? Uh, they both think… I think Lotus started this. They think that the year 1900 was a leap year.
00:55:31.000 --> 00:55:34.000
And it wasn't. Do you know the rules for leap year?
00:55:35.000 --> 00:55:46.000
I do, um, uh, and when you come upon a date that is, uh, 00, that is NOT.
00:55:46.000 --> 00:55:49.000
a, um… Debate.
00:55:49.000 --> 00:55:52.000
Well, here's the rules, in case anybody's wondering.
00:55:52.000 --> 00:55:58.000
Uh, if the year can be divided by 4, evenly divided by 4, then it's a leap year.
00:55:58.000 --> 00:56:13.000
Unless the year is evenly divisible by 100. So, like you said, a year that can divide by 00, or that ends in 00, is not a leap year. So, that's true. So, like, the year 1900 was not a leap year. The year 2100.
00:56:13.000 --> 00:56:15.000
Uh, will not be a leap year.
00:56:15.000 --> 00:56:24.000
Uh, that rule has another UNLESS if the year is divisible by 400. So, for example, the year 2000.
00:56:24.000 --> 00:56:26.000
That was a leap year.
00:56:26.000 --> 00:56:35.000
Right? So, you said a year ending in 00 is not? Well, the year 2000… the year 2400 will not… will be a leap year.
00:56:35.000 --> 00:56:46.000
So this leap year calculations are kind of done like this, so that the calendar stays aligned with the real solar orbit of the Earth.
00:56:46.000 --> 00:56:57.000
Uh, because, you know, we have, like, leap seconds and all that kind of stuff. It has to do that. Otherwise, we'll be celebrating Christmas in July, which, I don't know, maybe not be so bad.
00:56:57.000 --> 00:57:10.000
But, um, the Sony PlayStation… Um, uh, it, it treated in, in, on February 3rd, 2010, uh, it treated 2010.
00:57:10.000 --> 00:57:11.000
As a leap year.
00:57:11.000 --> 00:57:13.000
Even though it wasn't.
00:57:13.000 --> 00:57:25.000
Right? Because 2012 would be a leap year, 2008 would be a leap year. 2010 was not a leap year, but the Sony PlayStation treated 2010 as a leap year and bricked itself, trying to process a non-existent February 29th.
00:57:25.000 --> 00:57:30.000
I think that's, uh, kind of funny. Uh, the Zune, you remember the Zune?
00:57:30.000 --> 00:57:32.000
I do.
00:57:30.000 --> 00:57:36.000
The, uh, the, uh, Microsoft… that was Microsoft's answer to the iPod, wasn't it?
00:57:36.000 --> 00:57:43.000
a little music player that, uh… it was, like, twice the size of an iPod and nowhere near as cool.
00:57:43.000 --> 00:57:53.000
Um, it had a leap year calculation bug. The internal clock driver froze every, uh, on every 30 gigabyte Zune.
00:57:53.000 --> 00:58:01.000
worldwide on December 31st, 2008, and it stayed frozen until the clock ticked past midnight.
00:58:01.000 --> 00:58:12.000
So, it's New Year's Eve, you're celebrating with your friends, you got this music playing, you got Prince playing, uh, 1999, everybody's having a great old time, and the music player stops.
00:58:12.000 --> 00:58:22.000
It was the Zune, uh, uh, 2010 bug. Anyway, uh, that's, that, that's my list of, of, uh, bugs that.
00:58:22.000 --> 00:58:24.000
Should have been avoided.
00:58:25.000 --> 00:58:32.000
Um, I, I hoped you learned something. I, I think, I, I think we actually taught Wolf something today, so that's always a good day.
00:58:34.000 --> 00:58:40.000
Uh, yeah, some cases I had never heard about, but, um…
00:58:37.000 --> 00:58:39.000
Yeah.
00:58:40.000 --> 00:58:43.000
A lot of these.
00:58:43.000 --> 00:58:48.000
Choices that people made were.
00:58:48.000 --> 00:58:54.000
What do I have to do that I don't have to fix it?
00:58:54.000 --> 00:58:58.000
Um… And that's not right.
00:58:58.000 --> 00:59:00.000
That's what I feel.
00:59:00.000 --> 00:59:01.000
Yah.
00:59:02.000 --> 00:59:04.000
Yeah, it's, I mean, you know.
00:59:05.000 --> 00:59:08.000
Life is a series of compromises.
00:59:08.000 --> 00:59:13.000
But there are some compromises you should really be careful about, right?
00:59:13.000 --> 00:59:17.000
Yeah, I have feelings that.
00:59:18.000 --> 00:59:28.000
I want to write software such that that software is fixable.
00:59:28.000 --> 00:59:33.000
Um, by every programmer going forward.
00:59:33.000 --> 00:59:36.000
Not just me.
00:59:36.000 --> 00:59:40.000
Um… Yah.
00:59:39.000 --> 00:59:47.000
Well, see, you raise a… you raise a really interesting point, and this is one of the things I admire about you, the way… the way you said that. You want to write software that is fixable.
00:59:47.000 --> 00:59:52.000
You're implying that you know that your software may have bugs.
00:59:53.000 --> 00:59:59.000
But you want to make sure, above anything else, you want to make sure that that software is maintainable by other people.
00:59:59.000 --> 01:00:00.000
I do.
01:00:00.000 --> 01:00:02.000
That's admirable. I like that.
01:00:02.000 --> 01:00:15.000
Yeah, that is part of integrity, transparency, honesty. I absolutely don't want to do a job.
01:00:05.000 --> 01:00:07.000
Uh, it's part of.
01:00:07.000 --> 01:00:08.000
Yes.
01:00:09.000 --> 01:00:10.000
Yes.
01:00:15.000 --> 01:00:19.000
where I have written the code in a way that.
01:00:19.000 --> 01:00:22.000
It's a lever only I can pull.
01:00:21.000 --> 01:00:26.000
Oh, yeah. Yeah, under the name of, uh, job protection.
01:00:22.000 --> 01:00:36.000
uh… Yeah, that is… Yeah, that is a form of job protection, but it's totally counterproductive because it means your job going forward.
01:00:29.000 --> 01:00:30.000
Yah.
01:00:32.000 --> 01:00:33.000
Sure.
01:00:36.000 --> 01:00:41.000
is you're gonna pull that lever over and over and over again.
01:00:41.000 --> 01:00:42.000
Right.
01:00:41.000 --> 01:00:49.000
I don't want to pull that lever. I want to write the software anybody can own.
01:00:48.000 --> 01:00:49.000
Yes.
01:00:50.000 --> 01:00:54.000
That's good, that's good. Alright, why don't you take us out.
01:00:54.000 --> 01:01:05.000
Well, first of all, thank you so much, Jim. And as always, total pleasure podcasting with you.
01:01:05.000 --> 01:01:12.000
For all the listeners, the show notes contain Mastodon addresses for Jim.
01:01:12.000 --> 01:01:18.000
for me, and for the show. We absolutely love feedback.
01:01:18.000 --> 01:01:24.000
You can send feedback to the email address, feedback@reddit.com.
01:01:24.000 --> 01:01:31.000
runtimearguments.fm if you need to get to the website.
01:01:31.000 --> 01:01:41.000
It is not SSL. It is, um, HTTP colon slash slash.
01:01:41.000 --> 01:01:45.000
runtimearguments.fm.
01:01:45.000 --> 01:01:57.000
Um, absolutely. Tell us what you thought of this episode. Tell us what you thought of ANY episode that you have listened to and have thoughts on.
01:01:57.000 --> 01:02:09.000
Um, especially the one we just did, uh, earlier about leaning your ladder, because I really want to hear thoughts about that. Um, our show comes with a transcript.
01:02:09.000 --> 01:02:18.000
Which you can see in the show notes, or maybe by going to the website for this episode.
01:02:18.000 --> 01:02:23.000
Um, we want to hear what topics you care about.
01:02:23.000 --> 01:02:27.000
Are there things you'd like us to talk about?
01:02:27.000 --> 01:02:33.000
Did we go too far on this episode? Did we not go far enough?
01:02:33.000 --> 01:02:34.000
Um.
01:02:34.000 --> 01:02:49.000
Do you like How Was Your Week? Did we spend too much time on it? Not enough! Um, it has absolutely… Been a pleasure for both of us. I think I speak for Jim here as well.
01:02:47.000 --> 01:02:49.000
Yes.
01:02:49.000 --> 01:02:57.000
Um, talking to you all about things we think are important. Um, I just hope.
01:02:57.000 --> 01:03:04.000
that, uh, at least occasionally, we find the topic you think is important.
01:03:04.000 --> 01:03:15.000
Um, and uh, until then, uh, my feeling is, thanks so much for listening and getting to the end. I can't wait to talk to you next time. Jim, what about you?
01:03:15.000 --> 01:03:25.000
Uh, thanks again, uh, Wolf, for podcasting with me. It's always a pleasure, and thanks to the listeners for listening, and uh, hey, have a great couple of weeks. We'll see you next time.
01:03:26.000 --> 01:03:30.000
Awesome. All right. See you later, everybody.