Full Transcript for Episode 10110111: More jazz than binary

had to, I saw like, surprise faces what's happening? Did some No, I Allison was just I was just being like, is this a thing in Canada? Like, has the government ever shut down? And then I all I found was an article that was like, how we what we have in place so that we, this never happens here? I haven't read the article yet. But just because I was like, I don't know if I've ever heard about this happening. But that doesn't mean it's never happened. It's just Yeah, continue. Gary And the definition of like, like essential employees is pretty slim, at least in the areas I work in. So like, every civil servant I know is like, you know, I don't know, I'll figure it out. Like, are you kidding? Like, this is ridiculous. But the result was we we there was a conversation around the week like, well, can we launch any earlier than Thursday? And there are no technical reasons. But there were like real reasons like, Well, yeah, I mean, the people that change the DNS are set to do it on Thursday. And the people that like handle the SSL changeover and all that stuff are scheduled for Thursday. Like, I don't know if we can get them to move on a Tuesday night to make it happen on Wednesday, or Monday, let's make it happen on Wednesday, and all that mess. And so, we started launch at 2pm. We were supposed to do DNS cutover, at 3pm. But it didn't happen until like 7pm. The database search replays took a lot longer, because there was a requirement to log all the changes in the DB Search and Replace even though we had a backup from right before we did the search replace. So that's it. It's done. It's done. Yeah. And I walked downstairs at eight o'clock and was like, Yes. And 830 I got a message on my cell phone like, hey, there's like one tiny thing. Can you come take a look at? I'm like, Are you kidding? And I did and I was there till 10 o'clock. But oh, no, it wasn't tiny at all. That was it was pretty tiny. Like it was just a matter of like, I was moving very slowly and deliberately because I knew that I was just like, totally tapped out. Didn't want to make like a really dumb mistake can be like, Oh, I took the site down. Hold on. I'll bring it back up. One moment, please. That'd be a really bad look at that point in the day. Yeah. Yeah. Definitely broke some protocols on deploying to master but whatever. It's fine. Chris Did it? It did well. Turns out that I'm going to be going to bed camp, which is Bay Area Drupal camp. Nice. Allison I was just like, if they're already calling it bad, bad why? Chris It's, it's being hosted at the Pantheon pagoda in San Francisco. So my manager had suggested and it's in November, middle of November. If anybody is listening to this and going, you can say hi. Maybe it'll be on your team. I guess. It's a hackathon today hackathon, I guess and yeah, I'm doing we eat since it's at the pagoda. My managers like we should see We can get people who are interested to come out to, you know, do it. And it'll be my first time at the official Pantheon headquarters. Gary That's fine. Does it require like an ID and all that kind of stuff to get? I mean, probably not during the, Chris you know what? No idea. Allison Figured out. When in November Is it is it mid around Thanksgiving? Chris 1819. Allison It might be in the Bay Area later on in the month. So yeah, I'll be going to bed camp, but like, Well, Chris siding is always fun. I'll be I'll Chris be flying out the 17 flying to San Francisco 17th. We'll be flying out on the 20th. So the weekend before basically, that makes sense. Yeah, I didn't even think about Thanksgiving timing at all, which is interesting that they would have timed it the weekend before Thanksgiving, but who knows? Whatever. Allison Better than the weekend? Gary It's pretty regional though. Right? I would think. Yeah. So Chris there's, there's, I mean, it's basically a local Drupal camp. It just has a cooler name. And, and I don't think all of the Drupal camps are necessarily hackathons. This one aimed specifically at the Gary Drupal camps are bad. Chris Yeah, on the on the Bay Area, Allison it's kind of exciting to go to the headquarters. Chris Yeah, and it'll be my first Drupal event and, and all that sort of stuff. So. So this is tangentially related to Gary's NASA announcement, because the NASA website was migrating away from Drupal and I will be entering into I mean, not entering necessarily, like I have kind of been in the space vaguely from from the periphery for a while, I thought Gary you were gonna finish that sense that entering like, I'm not committing myself to the codebase in any way, shape, or form. Chris I'm definitely not doing that. But I can answer Drupal questions in as much as they are actually questions about composure. Allison And someone reached out and asked me about Drupal related things. And I don't even remember what the questions were, because that's how much I didn't understand what they were talking about. And I had to write back and it's just like, This is not my wheelhouse, but I don't even have people to refer you to. Also, they weren't willing to pay anybody for a consultation. So I was just like, I don't I especially don't have any but Chris I definitely don't have anybody to refer you to. Allison But I like I don't know, I referred them to like online resources. Like maybe these will answer these seemingly, Chris or Giza thing I think. Allison Check out these docs. Um, I went to WordCamp Vancouver. Gary Yeah, yes, I want to I want to report on that. Allison Um, it was good. It was good. Um, classic Alison introvert workcamp chatted with like a few people and then quickly retreated into my own world making. The talks I went to were really good. Overall, I kind of pieced out later in the day because I was just Chris that post lunch stretch is pretty rough. Allison It was hard and like I was really interested in a talk but it turns out that one of the speakers couldn't

make it because he didn't get his passport in time or something. So then he recorded his talk and I was like, let's find all like listen to the recorded talk but he was just reading off slides and I was like and then so then I was like out in the hallway off because I was just like I can't listen to someone resigns right now I'm just gonna fall asleep and I went out in the hallway and then was like I'll have another hour before another talk starts I was just like I might just call it a day and wander around the city instead. So that's what I did then it started raining and I was like yeah that seems right for Gary ya appropriate are pretty on brand Allison but it was nice just to walk around a bigger city and I don't know just chill out and Gary just work camp do pretty well with dietary stuff at lunch. Allison You know what I was pretty impressed because not only did they have like the separate options but they like kind of secluded it in a different area. So we didn't have to wait in the same line. Yeah, which sounds so petty but I was like, Look how long that line is. And I was like sweet the vegetarian line is like so so short. Gary I don't know why this reminds me of that. But there is this at WordCamp us at lunch one day there was this soup. That was was a vegetarian option and it was a butternut squash soup but not Like, okay, great. Love that. But it was like sweet like, oh, like almost like, sweet or like, nope, like a sugary, sweet. Oh. And it was it was bizarre. It was like the first bite you were like, Oh, this is interesting. By the third bite, you're like, I don't know that. But I ate it because I was like, I need to keep my stuff going. Allison That's how I felt at lunch. I was like, The lunch was delicious. But I was like, I have like, I was hungry, but not that hungry. And I was like, I need to eat otherwise, this is not I'm just gonna crash hard and not even make it back to the ferry terminal. Gary I had broccoli cheese soup for lunch yesterday thinking like, that's gonna be like a big bowl of it. Like that's gonna give me you know, the energy I need to get through. Allison That's like nap city. Gary What are you doing? Well, well, I was pretty. I was extremely amped yesterday, like I was, I was out of bed at 4am. Like, wide eyed. But yeah, then two o'clock turned into three o'clock. And we were just waiting for like DB stuff to happen. And that turned into like five o'clock. Like, if I put in this call for three hours, like, Oh, you? Oh, that's horrible. There were there were like 50 people that really 31 I think was most of the 30 or 31 people on this call between these hours of 2pm. And when we finally were live at 8pm Oh, all right. And a lot of it was like, silent. And people coming every 15 minutes for like status updates. Because this, we had this checklist for yesterday, all the steps that have take place. And it was I think it started like step 104 I think there were like 109. And three of them were mine, like three of them were things I had to do. Allison Are they like super official, like a NASA countdown kind of steps were there Gary last summer. So some of them were some of them there actually were like, like the DNS cat over requires someone to suggest, like, we've reached, you know, all this other, we've done all this other crap, that's probably what was said, but something like that. So we're ready to do DNS cutover, I suggest we, you know, implement the DNS change, and that's from a contractor. And then the civil servant has to agree. And then the person doing the DNS cutover can log both those names and make the change. And we had three domains that were being cut over. So that had to happen three times yesterday, or so six people to speak to make that happen. That was really official, I'm trying to think whether it really official ones there were. I mean, some of the stuff that was like, some of it could kind of happen in parallel. So once the DB update was changed, there was like, you know, on local personal machines, like update your host file and, you know, verify things are loading. The DNS stuff on the NASA side is a little tricky, because NASA, you know, manages the DNS. So making the change it was it was actually live for the public before was live internally on VPN. So that was problematic, just from the sense of like, can't do single sign on until it's available in the VPN. So there was a little bit that had to wait for the DNS to propagate internally, which took longer than externally. And those kinds of silly things, I think, I think the DNS was the only official official one. There were several where it was like, the question was raised, like, Are there any objections to, you know, entering into next step, but I had a couple of CLI commands that I ran, and once it to be served from a place was done, I could do that without really asking permission. That was just the requirement of this other stuff happening. We had a DDOS last night. Pretty big one. Cool. By department, like, Allison you're like, that's watch, Chris what do you have in front of the site to to mitigate or handle DDoS is like, I actually have no idea. Yeah, cool. Gary Like when I say it's like, not my department, like, presumably there's something? Oh, yeah. I mean, there's there's definitely infrastructure for it, because we were still like, like sub one second response at 95/95 percentile. What was happening? So 99th, we saw some, some delayed responses, but it was, you know, Yeah, apparently, it was fine. I don't know. I mean, it wasn't enough for anybody. I don't know how to get a hold because my phone was off. So I mean, what would I do? Like, having a DDoS? You're not going to be much Oh, yeah, exactly. Like, I'm not the person that has anything to do, which is cool. I mean, there's the nice part of having the structure is that there are people that are like there to, you know, handle and mitigate those kinds of things. And I am not one of

them. Yeah, we had some like minor like, Oh, hey, here's some redirects that are pointing to the, you know, the beta domain and then that kind of silliness. Um, It's just content hours this morning that, you know, our work confused or misunderstood or whatever. And, you know, trying, like my stuff isn't working the way I thought it should or it's missing or Allison that feels kind of par for the course. Just, yeah, we have Gary like 450 people in, in the NASA website, like managing their own content. Everybody from like, you know, it is yeah, for a site of the size. It is a lot. Yeah, yeah. And there's, I mean, there's some, there's some guardrails. But there's also, you know, there's some people like an editorial level that are, you know, the calm side that can do whatever they need to do on anybody's content. And sort of have the final say, and in making sure the quality is there and whatnot. So you know, that many people, you're gonna have some conflict, decisions, and it's, and who owns what, what path? That's fun. What content should be at this path? Chris Yeah, that sounds awfully familiar. Who owns what thing? Yeah. Gary Well, it doesn't matter from a development perspective. Chris We have a lot of things that matter from a development perspective. Gary Yeah. Yeah. And like, we, I was like, we're hitting a code freeze on like, a week before this thing. Knowing full well, like, we're gonna have to pause after that. But we targeted the code freeze a week ago on Thursday. And then Monday, we had a couple of releases that were minor, you know, Tuesday, there was some stragglers stuff like, oh, we need this tracking code, or whatever that needs to be in there. Chris When you? And I say you broadly, because it might not be you personally, Gary, but maybe it is you personally, Gary, decided that there should be a code for use. Yes. How is that communicated across teams? Gary Yeah, that's a great question. Um, Chris I have a personal vested interest in this. Gary Yeah. It sort of works this way. So we have a loose network of people that we know, are known to be contributing to the code base. So there's the primary development team, there are six or seven other teams that have committed code in different ways or are and that the primary dev team is supporting in some way, some kind of communication with. So we have office hours from time to time, like weekly from time to time, like literally on a schedule. It just, Allison they just need to pop up and surprise me sounds so like loosey goosey. I was just like, it's probably Gary there is totally a schedule. I don't I'm always surprised by it. And never mind that it happens every single week at the same time. I'm like, is that today? Yeah, it's still Tuesday. So that's one one communication Avenue, but also those office hours, backfill like the notification channel. So it's like, these are the people that we can notify in teams, because they've been to office hours, they're gonna see this, but we can also pull that out and send an email to all those people that have attended office hours and say, Hey, this is happening. We on the project, in general, there's also a person whose only job is communications with mostly editors, because there's so many, so many more of them. But I mean, pretty much anything we need. So, you know, like yesterday, there was a point where it was like, you are no longer allowed to log in, and people like, I'm having trouble logging in. Well, should you not read the message? Like it's, you know, that's also par for the course. Yeah. And then that same person, you know, drafts and sends out the message that we're live, but only after the civil servants have reported to their management, that things are live and their management looks at it and says, Yes, leave it live. I like we had a rollback plan in case they were like, No, but I mean, there's going to be like, if we have to roll back at this point, like, this is going to be an all nighter to roll back, you know, I mean, it would have worked, but it would have been miserable for all of us. So, and nobody expected that. But yeah, so yeah. So to answer your question, like we have a list of people that are contributing and show up for office hours that we communicate with, also, no, I mean, that's just what I mean, we probably have 25 Pull Requests open between Thursday of last week, and this week that I haven't looked at that I'll get to next week sometime. Yeah. So I mean, if there's a government Chris shutdown, and you can't work on things, well, I can I can review code, you're gonna work on things until Yeah, you can't. Gary Yeah, but I mean, there were there were tickets I was monitoring when those pull requests open. Those were the ones I was chasing down and making sure they're deployed and QA was aware of and so the other stuff was noise and I use tags on pull requests pretty liberally. So I can just visually check and be like, oh, all the pink ones. These are the ones I'm working and you know, these orange ones are cool to sit there for a while. I've got like a WTF purple tank. It doesn't say WTF but in my head that's what it says. It's labeled merge later on But that later is like sometimes that later it's better than being like, merge with this. Yeah. So merge later sometimes like, that turns into a dialog. It's like, well, maybe you should break this pull request into pieces. Or maybe this is the wrong approach or the wrong code base for this. Could you do this somewhere else? Like, or not at all? Yeah, so all that all that silliness, also, is just part of it. So yesterday, we had like, No, it was it was funny, because our colleagues over at the science side where we're chasing a bit more, they weren't able to get to where they wanted to be, you know, just had some more demands from stakeholders that we didn't. So at one point, someone said, I'm really jealous of your lack of traffic in your repo right now, because they missed

it. And then they messaged me at like, one o'clock about a question on some language in some static strings that still mentioned, you know, the, it didn't talk about the beta site, but like something like during this testing phase? And so I was like, I don't know. So that one had to go through a bunch of other people that approved these language things. And like, Nope, it's staying that way. But I thought we were gonna have to have like a, you know, quick pull request there. But no, nothing. And so we went live. And there was one PR that couldn't be merged until we were alive. Because it would be redirect loop. So that was staged and ready to go. That was one of my three things. And then last night, I had two tiny ones that were just related to other silliness. And and I've had one today, we never deploy on Fridays, but with a shutdown looming, it doesn't feel like a Friday, so. Yeah. Yeah, it's, it's done. Allison Yay. Chris Yeah. Yeah. Gary Now it's done. And now the real work begins. Yeah. The, the cool part is that there is absolutely recognition that this is not like a one and done process and that there is, like, a healthy budget in place to keep people running. Yeah. Yeah. I mean, there's a backlog of features. But in addition to that, there's just there's bugs and there's accessibility misses. And there's mobile misses. And there's, you know, there's just things that need to be handled and, and the space is there for that. And the right people will continue on for that. So that's also very, very cool. Chris I got I got a new laptop this week. So what are for no for some, because the lot my old laptop, the screen died. I've had this before. See, yeah, the last time it happened was Aaron's laptop, which is actually my older older laptop. And like the screen, just like there's a certain point where the cable that connects the screen to the to the laptop, just like gets worn, and then the screen just starts like, like, glitching out basically. With her it like really glitched out like, it was like weird, staticky kind of stuff. And with me, it just sort of like gradually faded to white, which would then like, Go away temporarily, if I like closed the laptop a little bit and reopened it. But then it would do Allison this a strategy for a little while. It only Chris lasted like two days. I mean, it was like it was literally like, one day last week it started doing it. And I could kind of get around for around it for like two days. And I'm like, Yeah, I should probably do something about it. And then after that, it that was it. Like I really couldn't use it. So I took it in on on Tuesday, and it was gonna be $600 to replace the display. That laptop is also having Sticky Keys because it has the old butterfly keys. So it'd be $600 to fix the display and not the keyboard. And I'm sure, right, like, and so like the display needs to be, that's the whole top half and then the keyboard, I'm pretty sure they basically replaced the bottom half. So like at that point, what's the point of having, Gary like, what am I even doing? Right? Well, you're keeping the hard drive. Chris Right? So when when we went in last year when this happened to Aaron, they said well, or you could get a MacBook Air like the the bottom level of MacBook Air and we're like, Well, I'm gonna kind of like and so he didn't do it. And then later we kind of kicked ourselves we're not doing it, we're likely probably just should have done it. So I asked him like, what would it be like last time we had this happen? They they suggested that what would the trade end be? What would it what would the math look like? So they had a bottom level and one air that would have been Like, so it got $370 in credit and that one the bottom levels like 1099. So would have been like, a little bit more than what I would have been paying for the display or $400 More than that. I could get an m two, and I could make it black. So Transcribed by https://otter.ai