Defining Data Engineering with Maxime Beauchemin - Episode 3 का पूरा ट्रांसक्रिप्ट

grow. So 57 00:04:23.640 --> 00:04:27.500 the problem domain that 1 person needs to be capable of handling 58 00:04:27.800 --> 00:04:29.925 is growing larger as 59 00:04:30.544 --> 00:04:40.870 the expected delivery times are becoming shorter. So Yes. On 1 side, there's there is some sort of specialization, but we also expect for people to, to be kinda wider and cover, 60 00:04:41.330 --> 00:04:42.150 more and more 61 00:04:42.610 --> 00:04:44.710 fields or related, you know, specialties. 62 00:04:45.145 --> 00:05:00.295 So it's really hard for someone that's very specialized and that knows only kinda 1 aspect of software engineering or data engineering to be useful at a company. It's always a balance of going deep in some areas, but, kinda wide and and being able to do kinda full stack as much as possible too. 63 00:05:00.775 --> 00:05:20.264 1 1 other fact is, the idea that, you know, in data engineering, as in software engineer engineering, like, the the only constant is is change, you know. Things are changing very fast. The Hadoop ecosystem and the data landscape is definitely, like, diverging still. Right? We don't see a lot of convergence. Now there's more databases than there ever was. 64 00:05:20.645 --> 00:05:22.585 There there's more kind of pieces 65 00:05:22.965 --> 00:05:25.465 of technologies and platforms and frameworks 66 00:05:26.110 --> 00:05:34.690 and libraries. So there's this explosion of knowledge and code that we're essentially, like, trying to stay on top of as data engineers. 67 00:05:35.465 --> 00:05:50.450 And do you think that the DevOps movement that has sort of come to prominence over the past few years has had any impact on the discipline and the concept of data engineering as a whole. And I'm wondering if there are any particular kinds of crossover that you've seen whether as far as the philosophy 68 00:05:50.830 --> 00:05:52.690 or the tooling that's available. 69 00:05:53.175 --> 00:06:07.660 Right. So I think in terms of tooling, you know, companies share, all these pieces of infrastructure. So when you when you talk about DevOps, I'm not sure exactly kind of which part of infrastructure you'd be talking talking about. But but let's say 1 component might be stuff like, continuous integration, 70 00:06:08.440 --> 00:06:12.540 unit testing, kinda automating the work of developers 71 00:06:12.884 --> 00:06:14.504 and you know, data engineers are definitely 72 00:06:14.884 --> 00:06:21.705 just as interested in that as people in dev ops or any type of software engineering. 1 place where we see kinda 73 00:06:22.350 --> 00:06:36.074 like an historic kinda divergence and and things are starting to change a little bit is, the way the data that that we work with. So dev ops or, you know, ops group will have, typically an ODS or an operational data store, 74 00:06:36.375 --> 00:06:38.794 where they'll have, you know, time series databases 75 00:06:39.175 --> 00:06:43.680 or things like at Airbnb, we use something called Datadog. At Facebook, there's something called, 76 00:06:44.159 --> 00:06:46.740 ODS that was all about kinda real time metrics 77 00:06:47.360 --> 00:06:52.485 of, machines and performance metrics of, you know, anything related to to development. 78 00:06:52.865 --> 00:07:00.820 And, we're starting to and on the on the other side, you know, on the the data engineering side, we've been typically very focused on the data warehouse itself. 79 00:07:01.200 --> 00:07:05.380 In a lot of cases, kinda slower systems, kinda 90 days analytics, 80 00:07:05.745 --> 00:07:09.685 batch processes. And, you know, historically, it's been different kinda technologies 81 00:07:10.065 --> 00:07:14.410 for databases on both sides. We're starting to see more and more data engineers gonna get 82 00:07:14.970 --> 00:07:16.410 outside of of, 83 00:07:17.290 --> 00:07:21.790 kinda batch processing and get more into real time with technologies like Spark Streaming, 84 00:07:22.845 --> 00:07:24.544 technologies like like Druid. 85 00:07:24.845 --> 00:07:26.064 And, you know, slowly, 86 00:07:26.845 --> 00:07:30.224 we see, you know, DevOps people using the same databases 87 00:07:30.685 --> 00:07:31.890 as data engineers. 88 00:07:32.690 --> 00:07:38.310 And and 1 example of that at Airbnb would be, the use of, the Druid dot I o database, 89 00:07:38.850 --> 00:07:40.550 which is a a real time, 90 00:07:40.985 --> 00:07:48.505 distributed column store that works very well for, like, real time analytics or just like fast analytics, kinda fast paced, big scans, and, 91 00:07:49.090 --> 00:08:00.095 kinda being able to, crunch and, like, crunch a lot of data. So, so both the people in DevOps and the people doing more classic analytics and data scientists use, this database. 92 00:08:01.034 --> 00:08:04.414 And for somebody who is working in data engineering, 93 00:08:04.955 --> 00:08:06.095 I guess, how much 94 00:08:06.430 --> 00:08:20.195 interface do they have to the actual managing of the underlying infrastructure versus just having an operational team who provides those servers and deploys the services, and the data management team is responsible for building the pipelines and tuning the actual, 95 00:08:20.574 --> 00:08:22.035 services running on those instances. 96 00:08:22.430 --> 00:08:31.865 Right. So I think it's a factor of how big perhaps like the company or the the data team actually is. But we definitely see, at some point in time, some sort of specialization 97 00:08:32.165 --> 00:08:34.905 where at the very beginning, maybe as the first 98 00:08:35.285 --> 00:08:39.305 data engineer or the first software engineer with a data focus at a company, 99 00:08:39.620 --> 00:08:46.040 your first task is gonna be around, you know, setting up some infrastructure, getting, you know, things like Kafka and Hadoop, 100 00:08:46.580 --> 00:08:50.945 and, you know, Spark up and running. And you know, if there's only a few, 101 00:08:51.585 --> 00:08:54.005 data

engineers in the team, then there's probably 102 00:08:54.545 --> 00:09:04.985 gonna be, you know, distribution of of tasks where people will do a little bit of both. And over time, they they will kinda shake out in in a certain way that, 103 00:09:05.305 --> 00:09:06.445 people will specialize. 104 00:09:07.144 --> 00:09:09.245 Whether, you know, whether managing, 105 00:09:09.704 --> 00:09:11.404 deploying, and managing, and maintaining 106 00:09:11.865 --> 00:09:25.115 the data infrastructure is the role of the data engineer. I would I would argue that not that it's not the case in in a bigger company that, you know, other people that are perhaps a little bit more dev DevOps y, or people that are a little bit more focused on, 107 00:09:25.654 --> 00:09:26.154 infrastructure 108 00:09:26.455 --> 00:09:34.639 would would take on, those tasks in a company, and it's very different skill set in a lot of cases. So so, yeah, I think over time, 109 00:09:35.180 --> 00:09:47.605 data engineers will will tend to go and focus more on things like the data warehouse itself and the all the data plumbing and, you know, data structures and getting consensus as to how the data should be organized, 110 00:09:48.080 --> 00:09:51.060 and as for for for, for the company. 111 00:09:51.920 --> 00:09:55.595 And for somebody who's interested in getting started in the field of data engineering, 112 00:09:56.135 --> 00:10:05.110 what are some of the necessary skills that they should possess and what are some of the most common backgrounds that you see those people coming from? So so there's, different types of people. There's the people, 113 00:10:05.830 --> 00:10:12.204 the dinosaurs like me who came more from the business intelligence and a more traditional classic data warehousing 114 00:10:12.505 --> 00:10:29.950 field. But you'll need to if you're coming from that background, you're gonna need to be able to, to code or you're gonna need to develop, you know, the the skills around around source version control, writing code, and starting to, change your processes so that your daily tasks, look a little bit more like a 115 00:10:30.415 --> 00:10:33.155 classic software engineer. So so there's definitely, 116 00:10:33.775 --> 00:10:40.940 people would have to kinda recycle their skills. And the people that are coming from there usually are very strong at things like data modeling, 117 00:10:41.560 --> 00:10:46.940 and ETL and performance tuning. And, it's typically people that know very well how databases 118 00:10:47.399 --> 00:10:55.080 work. Perhaps people that were, you know, DBA in the in their previous life. And then there's all the people coming from from other other places, like just, 119 00:10:55.480 --> 00:11:03.420 new grads out of computer science. There's people coming from the field of data science that realize that they're perhaps more interested in building 120 00:11:03.955 --> 00:11:04.615 persistent structures. 121 00:11:05.315 --> 00:11:16.750 Right? People that that were in data science, but realize that they're more interested in doing engineering type work. So I would say patience is also probably a very important 122 00:11:17.130 --> 00:11:20.670 thing because a lot of the, you know, batch processing data pipelines 123 00:11:21.050 --> 00:11:22.190 can be pretty cumbersome. 124 00:11:23.115 --> 00:11:44.524 And there's definitely like a part of, data data engineering that is about kinda plumbing. Right? So data plumbing and managing the pipes and sometimes, you know, all hell breaks loose and you gotta get in there and and fix this stuff. So that's 1 aspect of the job maybe that that's a little bit less glamorous, you know, the the plumber aspect. But then there there's all tons all sorts of other aspects, like, if you want to automate your work, there's, 125 00:11:48.800 --> 00:11:56.740 like in data engineering as in any type of of software engineering, there's always tons of opportunities to create abstractions and create tooling 126 00:11:57.355 --> 00:12:04.095 where you can automate kinda your own work, as a as a data engineer and then build, say, services or 127 00:12:04.780 --> 00:12:32.995 systems or framework that, that do the things that you would, you would have done manually as a data engineer before. So for the data engineers out there who are, getting bored with the job or or that are thinking, you know, data engineering is is there's a lot of like, you know, data data pipeline writing and that's not very exciting. There's always an opportunity to to take the things that are redundant and and build services and and systems around it. And that's kinda what I've been doing over the past few years with with Airflow, and that's been super super, 128 00:12:33.375 --> 00:12:33.875 rewarding. 129 00:12:34.815 --> 00:12:43.750 Yeah. In my other podcast, we had a nice long conversation about the work that you're doing in Airflow. And since then, I know that it's become an Apache incubated project. 130 00:12:44.209 --> 00:12:58.680 Wondering if you can touch briefly on some of how it has evolved since the last time we spoke. I know that it had been using Celery as the execution layer. I don't know if that's still the case. It it is still the case and, I'd Yeah. I can talk a little bit about the the Apache, 131 00:12:59.300 --> 00:13:01.240 software foundation and and how, 132 00:13:01.620 --> 00:13:05.079 it it, you know, helps a project grow, but it also, 133 00:13:05.785 --> 00:13:06.605 really changes, 134 00:13:06.985 --> 00:13:12.125 the dynamics of the the project too. So I've been through that for the first time, you know, recently. 135 00:13:13.020 --> 00:13:30.264 And Airflow started as an Airbnb project, and we we had kind of full control over the project, and we, you know, would release whenever we were ready. And then at some point in time, we we were interested in joining the Apache Software Foundation because, you know, there's a lot of companies that that

consider 136 00:13:30.644 --> 00:13:42.695 it kind of a risk to, start running software that they don't have, any guaranteed control over. So what the Apache Software Foundation does as you as you start the incubation process is first, 137 00:13:43.235 --> 00:13:48.295 you you have to donate your code and your trademarks and your intellectual property, 138 00:13:48.680 --> 00:14:03.375 to the Apache Software Foundation, which is only gonna make sure that, you know, people are not gonna change their mind about sharing. So, you know, from that point on, the Apache Software Foundation owns your code and the trademarks and is has some responsibility around, 139 00:14:04.020 --> 00:14:11.800 some of the legal aspects of the of the project. Right? If someone was to try to sue us because, say, the name airflow has been used by some other company, 140 00:14:12.340 --> 00:14:35.524 or something like that, the Apache Software Foundation would would help us, would probably manage this litigation. You know, this is as much as I know about the the Apache Software Foundation. So maybe some of the things I'll say are not a 100 percent right. So hopeful hopefully, they they mostly are. But a part of the process also is to, the Apache Software Foundation make sure that, there's different people from different companies involved jointly, 141 00:14:36.145 --> 00:14:48.190 and they create what they call a PMC, which is, I believe a project management committee that has to be made of, you know, 5 or 6 or perhaps more committers that have kind of the same level and of involvement 142 00:14:48.495 --> 00:14:56.435 with the project. So I make sure that say, in the case of, for example, I don't know the whole story with Apache Storm, but at some point Twitter 143 00:14:56.750 --> 00:14:59.570 had developed Storm. A lot of people were using Storm. 144 00:14:59.870 --> 00:15:29.625 And at some point, they were like, okay. We're done with Storm. Now we're just gonna build this other thing called Heron, and we're not interested in contributing any more code to, Storm. So at that point, they, they gave the the software to the Apache Software Foundation, and other people from other companies created a little committee and, manage jointly the project together. From from a developer standpoint, it's also a pretty interesting thing because, like, right now the work that I do say on airflow is if airflow is owned by Airbnb 145 00:15:30.209 --> 00:15:40.214 and something changes as to my employment status with Airbnb, I might not be able to, to work on airflow anymore and I might have to fork the project or 146 00:15:40.675 --> 00:15:50.650 something of that nature. But with the Apache Software Foundation, you know, I I know that I'll I'll always be an airflow committer and have the same kinda relationship with the project regardless 147 00:15:51.110 --> 00:16:01.185 of of my, employment, which is cool. So I think also there's just kind of a a brand that's associated with it too. So companies have a lot more trust into, 148 00:16:01.725 --> 00:16:04.225 say, installing or running an Apache project, 149 00:16:04.529 --> 00:16:22.125 than they they would have, with just running like a piece of software that's developed at some other company. Yeah. Maybe a a few more things about the Apache Software Foundation. So so joining Apache has definitely kinda slowed down the pace of of change for the project. For and that I think that's part of 150 00:16:22.650 --> 00:16:23.230 a maturing 151 00:16:23.610 --> 00:16:27.230 project to kinda slow down and focus on things like stability 152 00:16:27.610 --> 00:16:33.904 and just making sure you have, like, a solid product that works for everyone. So the the pace of development 153 00:16:34.205 --> 00:17:14.330 slows down a little bit, but the quality of the releases goes up. It's been kind of a challenging for the the airflow project for us to come up with a release. I believe the last release was about, like, a little bit more than 6 months ago, and, people in the community are working on the next release, but for different reasons. Now it's like we have all sorts of code that comes from different companies, and that's all and that's that same, melting pot. And we have to make it work for everyone, and that can be challenging, especially the first release. But I'm hoping that, you know, with Thyme and with the community, we're gonna be able to, to come with, like, a steady release process pretty, pretty soon. Yeah. It definitely seems that having the stamp of the Apache Software 154 00:17:14.790 --> 00:17:23.635 Foundation, it's a boon to a project. And like you said, it does connote a certain amount of maturity and stability. And it's interesting how many of the different widely 155 00:17:24.095 --> 00:17:29.155 used projects in the big data and data engineering space have ended up under the umbrella of the Apache organization. 156 00:17:30.015 --> 00:17:42.185 Right. Especially considering kind of all the restrictions that come with it and the bureaucratic process that, goes along with going Apache. Right? So there's we have to use Jira, for instance, instance, and there was some limitation on how we could use, 157 00:17:42.985 --> 00:17:46.365 GitHub. And I was like there's this like, these old mailing lists that we need to use for, for legal reasons. So we're, 158 00:17:49.880 --> 00:17:54.060 you know, say, unable to use GitHub issues. We have to use Jira instead. 159 00:17:54.360 --> 00:18:14.590 And, you know, at the beginning of the project, that was a little bit frustrating to just have to step back and use some of the Apache infrastructure that that is really kinda decade old at this point. If you go sometimes you'll end up you do a Google search, and you'll end up on some Apache mailing list, you know, website where it looks like the CSS is from, you know, the 1900. 160 00:18:15.005 --> 00:18:18.385 But, but all of that, you know, I think it was a really good choice for

Airflow, 161 00:18:18.845 --> 00:18:21.405 and I think that it really contributed to the project's, 162 00:18:21.805 --> 00:18:23.390 growth because it attracted, 163 00:18:23.769 --> 00:18:33.385 you know, very talented committers and developers that were like, hey, I can be an Apache committer. Or some of them were Apache committer before on other projects, and then they can really get involved, 164 00:18:33.845 --> 00:18:40.159 and trust that that things are going somewhere. Right? If you if you say if I wanted to get involved, 165 00:18:40.539 --> 00:18:50.134 with, say, Luigi, which is a similar product to Airflow in some ways, I would have to kinda, you know, negotiate with with Spotify or somehow, you know, 166 00:18:50.434 --> 00:19:02.460 try to get them to, like, let me let me join the project and there might not be a process for that. There might not be any guarantees. With an Apache project, there's kinda always a way for a company to get involved and to become 167 00:19:02.835 --> 00:19:04.775 decision makers as part of the project. 168 00:19:05.315 --> 00:19:11.414 What do you see as some of the biggest challenges facing data engineers currently in the discipline of data engineering? 169 00:19:12.230 --> 00:19:23.255 1 of the new things is kinda all the real time data. So, you know, Spark Streaming, Tamsa, databases like Druid. So that's kind of a new stack for a lot of us and that's that's also 170 00:19:23.635 --> 00:19:25.895 new type of technology, new set of constraints, 171 00:19:26.275 --> 00:19:29.840 you know, so people adapting to things like beams, 172 00:19:30.140 --> 00:19:41.975 Apache Beam, which is the Google data flow, based on the Google data flow paper. And then not only knowing all the batch processing stack, which is fairly complex, right, with things like MapReduce and, 173 00:19:42.755 --> 00:20:00.645 Hive and, you know, schedulers like Airflow. Now on top of that, you you have to go and learn the the real time stack. Or perhaps it's in bigger organization, it's different people that will focus on these different aspects of data engineering. What else? So some of the challenges is, like, always always working, 174 00:20:00.945 --> 00:20:01.445 with, 175 00:20:01.825 --> 00:20:10.000 with data science has always been a challenge, and there's always this kind of push and pull as to, like, whether a data engineer should be building kinda horizontal 176 00:20:14.345 --> 00:20:21.290 engineering should go and work closely with a project team. So that would mean something like someone at at Airbnb might, 177 00:20:22.010 --> 00:20:23.310 be the the data engineer 178 00:20:23.690 --> 00:20:50.394 assigned to a specific product team, say, working on the new trips feature that we have, and build the the data structures for them. And, you know, the data engineer has been always gonna have the pivotal point as to, like, whether they work on horizontal or vertical things. And sometimes it's hard to kinda to please everyone being in that pivotal point, and there's all these forces pulling you in different directions. What else is challenging? I mean, just keeping up with the diverging data landscape and ecosystem 179 00:20:50.855 --> 00:20:55.755 has always been a challenge. Right? There's so many there's a new flavor of of database 180 00:20:56.640 --> 00:20:59.380 every day, and there's just so many packages 181 00:20:59.760 --> 00:21:01.620 and libraries and frameworks, 182 00:21:02.080 --> 00:21:05.220 that you have to stay on top of. And that's always been 183 00:21:05.575 --> 00:21:12.875 exciting, but but kinda challenging too. Right? If you stay for too long put at the same place or kinda same stack, you can be kinda pigeonholed 184 00:21:13.175 --> 00:21:18.260 in a certain role or on a certain stack. So, those are some of the challenges I can see. 185 00:21:18.980 --> 00:21:31.335 The the main challenge for a data engineer, I think, that I see is operational creep. Right? So I think it's true of of a lot of careers in information management in general and software engineering, 186 00:21:31.790 --> 00:21:37.650 But it's really easy over time to build more and more more things that you have to maintain 187 00:21:38.030 --> 00:21:41.535 and to go from a place where you start perhaps, like, doing 90% 188 00:21:48.815 --> 00:22:11.210 And then And then over time, as you stay with the company for, you know, 1 year, 2 year, 3 year, it's really easy to pivot and realize at some point that you're spending, say, 80, 90% of your time, like, just kind of fixing stuff and doing some some fighting fires and not adding a lot of value because you're just kind of the guy that keeps the things, running. So it's important for people that 189 00:22:11.590 --> 00:22:36.235 identify this to to go and invest on tooling and on, you know, refactoring and paying off that technical debt so that they can stay challenged and and and so that they can keep creating as opposed to, you know, just kinda fixing stuff that they've built in the past. And you touched briefly on this, but how much analytical knowledge do you think is necessary for somebody who's working as a data engineer? 190 00:22:37.174 --> 00:22:40.794 So I think you you need to have definitely like this analytical instinct. 191 00:22:41.100 --> 00:22:51.635 Right? And that is, you know, vital for for data scientists to have, peep People have called that product sense too. I'm not sure if that's what you're you're talking about here, but, you know, you need, say, 192 00:22:52.035 --> 00:23:00.820 as an analyst, as a data scientist, you absolutely need to have that product sense and have that intuition as to just being being really curious 193 00:23:01.120 --> 00:23:22.809 and kinda prone to dig in until you find the answer you're looking for. So I would say, that is critical for analysts and and data scientists. I would say it's really important for for data engineers too. Though, you know, for a data engineer,

there's that other urge, which is the the urge to build infrastructure and to to build things that, are there to stay, that you have to balance, 194 00:23:23.110 --> 00:23:25.450 with with the more sometimes, like, ephemeral, 195 00:23:26.485 --> 00:23:28.505 analytical urges you might have. 196 00:23:29.044 --> 00:23:32.825 And also how much sort of statistical analysis or understanding 197 00:23:33.205 --> 00:23:36.750 of some of the different mathematical principles or probabilistic principles 198 00:23:37.370 --> 00:23:39.630 are necessary for being able to properly manipulate 199 00:23:40.490 --> 00:23:46.665 the data so that it is in a shape that is accessible for the people who are doing the end analysis on it. 200 00:23:47.045 --> 00:23:54.059 I think this part I would call so that we call this data modeling. Right? And that which is very different from statistical analysis, 201 00:23:54.840 --> 00:24:07.169 like knowledge about stats. Right? So, let me talk a little bit about data modeling. So data modeling is is not a perfect science. There's been all sorts of books written on it. Right? There's this concept of, you know, star schemas and snowflake schemas 202 00:24:07.789 --> 00:24:32.740 and Ralph Kimball and, Bill Inman are people that wrote about that stuff in in the in the nineties. Right? And those books in part, you know, still some aspects of those books are very relevant still today. Some aspects might be a little bit less relevant than they than they they used to be. But data modeling for a data engineer is, like, kind of a a core skill. Right? It's how how are you gonna structure your tables, your partitions, 203 00:24:33.174 --> 00:24:35.914 where you're gonna normalize and denormalize data, 204 00:24:36.215 --> 00:24:37.034 in the warehouse. 205 00:24:37.335 --> 00:24:39.595 Are you gonna be able to say retrieve 206 00:24:39.895 --> 00:24:40.395 attributes 207 00:24:40.775 --> 00:24:52.385 of say the customer at the time of the transaction versus the the most recent attribute of the customer? So how do you model your data so you can ask all these questions? And data modeling for analytical purposes 208 00:24:52.765 --> 00:24:56.144 is very different than, say, data modeling for OLTP 209 00:24:56.605 --> 00:25:00.760 types type of applications. So OLTP is just, you know, classical, 210 00:25:01.380 --> 00:25:02.600 kind of databases 211 00:25:02.900 --> 00:25:04.020 structure for, 212 00:25:04.500 --> 00:25:13.205 building building simple software. So data modeling, I would say, is very important. Now talking about, knowledge about stats and statistical analysis, 213 00:25:13.505 --> 00:25:17.990 I would say not very much so. And I would even argue, you know, and I'll probably, 214 00:25:18.630 --> 00:25:22.890 you know, get people outrage on that but I would say it's, it's also not very important 215 00:25:23.304 --> 00:25:28.845 or not as important as 1 might think for data science. Because a lot of what we do in analytics 216 00:25:29.225 --> 00:25:31.485 is just counting things, right? And 217 00:25:31.990 --> 00:25:38.649 trying to figure out, like, are we doing better than we used to? And, you know, what is the growth rate and, you know, 218 00:25:47.289 --> 00:26:02.615 counting things. The stats that we use and the stats that I worked on more recently that are kind of very important, but that that we have extracted out for pretty much all companies working on an experimentation framework. Right? So most modern companies do experimentation, 219 00:26:03.795 --> 00:26:04.775 with their users 220 00:26:05.300 --> 00:26:15.565 and that usually, you know, people will call experimentation like an AB testing framework. And it's really important then for a company or especially a web company to be able to run a lot of experiments. 221 00:26:16.264 --> 00:26:17.325 And those experiments, 222 00:26:18.105 --> 00:26:26.220 you know, to usually usually, you'll have different treatments and a control, and you wanna see whether there are the changes in behavior are statistically 223 00:26:26.520 --> 00:26:27.020 significant. 224 00:26:27.480 --> 00:26:37.600 But say at Airbnb, I was involved in in building some of the data pipeline for our AB testing framework named ERF. I believe there's some papers out there or some, 225 00:26:38.240 --> 00:26:49.775 probably some videos and presentations that some of my colleague have done over the past year or 2. But that part of stats has been commoditized at Airbnb where now you can go and create an experiment, 226 00:26:50.315 --> 00:26:59.870 and deploy it and consume the the results. And of course, you need to know what a p value is and you need to know what confidence intervals are from just a consumption perspective. 227 00:27:00.425 --> 00:27:03.005 And you don't necessarily need to know and understand 228 00:27:03.385 --> 00:27:15.299 all of the aspects of the mathematics behind it. So I would say, you know, the stats work in a lot of cases and analytics is a little bit overrated. Though, you know, in some cases, it's really important for data science. But I think it's sometimes, 229 00:27:15.600 --> 00:27:18.179 it's often overrated and it's also 230 00:27:18.595 --> 00:27:28.649 often very well abstracted out too. So you can easily do some very complex things by using very simple libraries and functions, with clear APIs. 231 00:27:29.110 --> 00:27:37.450 And to make sure that the data that you're working with is of sufficient quality, what are some of the considerations that you need to be aware of when you're establishing new data sources? 232 00:27:38.215 --> 00:27:40.075 So data is kind of a jungle. 233 00:27:40.375 --> 00:27:46.235 You know, there's there's all sorts of data out there and as a data engineer, I didn't I didn't talk about that workload 234 00:27:46.590 --> 00:27:50.610 before or that that burden, which is, data integration. Right? So 235 00:27:50.910 --> 00:27:57.635 you need to go pretty often

and go and fetch data coming from, you know, different partners and different companies. 236 00:27:58.255 --> 00:28:06.240 And you need to integrate your, say, your referentials, like your list of users or your your list of accounts and transactions, 237 00:28:06.940 --> 00:28:10.400 with some external service providers. And that's always, 238 00:28:10.940 --> 00:28:44.990 that I think that's as challenging as it's ever been. Like, we thought at some point that, kinda the b 2 b, you know, data flow would get kinda fixed over time and that that would it would become easy that you could just kinda have some easy APIs that would sync up your, your data with, say, Salesforce or some of these service providers. And data integration is is, you know, as challenging as it's ever been. I think going going back a little bit more to your question as to data quality, right, So in some cases, we wanna get data from other systems from And we're concerned about data quality because we don't have control, 239 00:28:45.450 --> 00:28:59.500 around the instrumentation of how this data is generated. The the first thing is probably to make sure you you get things right where you're actually generating the data. Right? So say, most web companies will have some sort of what we call an instrumentation, 240 00:29:00.440 --> 00:29:06.505 framework, meaning that engineers that wanna track certain actions on on a website or on mobile, 241 00:29:06.885 --> 00:29:22.125 will have a a way to put kinda tokens and little trackers on the site to be, like, when someone hits, the booking button at Airbnb, make sure we track that. And I think it's really important to have what we call, you know, schema enforcement as up stream as possible. So that 242 00:29:22.505 --> 00:29:30.120 means as you generate that data, you want for that data to be to come to life, you know, to be originated with as much quality 243 00:29:30.500 --> 00:29:36.760 and have all the dimensions and all the metrics that you need from that moment on. And that can be really challenging, 244 00:29:37.060 --> 00:29:38.760 to do that. Now when, 245 00:29:39.095 --> 00:29:41.434 when you integrate with external, 246 00:29:42.215 --> 00:29:42.715 sources. 247 00:29:43.095 --> 00:29:51.970 Right? So if you you're fetching data from Salesforce API or, you know, in our case, we we use Zendesk, which is a kind of a ticketing 248 00:29:52.270 --> 00:30:06.470 system provider. And, you know, when we get data from them, sometimes you you don't know what to expect. Right? They might change their API overnight and let not let you know, or, you know, a certain referential that you might have some very specific rules on, 249 00:30:06.850 --> 00:30:12.855 might change over time. So, you know, in in batch processing and in stream processing, you you can, 250 00:30:13.795 --> 00:30:18.455 embed some data quality check. So that would mean that as you write your pipeline, 251 00:30:18.830 --> 00:30:22.930 you might have an idea of problems that may occur in the future 252 00:30:23.390 --> 00:30:27.924 and set up some some alerts and some break points saying, hey. If there's 253 00:30:28.225 --> 00:30:30.804 certain variables that go over a threshold, 254 00:30:31.105 --> 00:30:35.684 don't load that data into production. And if there's a value I don't recognize 255 00:30:35.985 --> 00:30:36.690 for a certain 256 00:30:37.169 --> 00:30:49.595 field, you know, send an alert. So that's really, very much case by case. And in a lot of cases, it's it's about kinda developing a, immune system over time. So that means, at first, you work with the API, 257 00:30:50.055 --> 00:31:08.885 you take for granted that it's not gonna change, you build your pipeline and 1 day, you know, something happens and the the feed, you know, there there's a bug of some of some sort or the data is not what it should be, then you might go and add some more checks and, like, kinda develop some mechanism to to alert or to, 258 00:31:09.285 --> 00:31:13.260 to prevent from this data from from making it to, to production 259 00:31:13.720 --> 00:31:16.860 before someone looks into it and and fixes the issues. 260 00:31:17.160 --> 00:31:19.080 1 pattern that that we do a lot, 261 00:31:19.640 --> 00:31:23.365 at Airbnb on airflow is, you know, as we do batch processing, 262 00:31:23.665 --> 00:31:41.425 very often we'll stage the data. So that means, we bring the data into a staging area. You can think of it like, you know, if you have a warehouse, that would be like where all of the trucks kinda drop the things before they get sorted out and brought into the warehouse. So we have this persistent staging area where we load the data and then we'll, 263 00:31:42.205 --> 00:31:45.745 we'll perhaps, like, run a set of checks on it. So stage, 264 00:31:46.045 --> 00:31:47.105 check, and then 265 00:31:47.440 --> 00:32:07.730 exchange. And the idea of exchanging is to bring that data into, into production or into a validated space. So, with airflow, we use a we made this little sub DAG. So it's like a kind of a construct that we reuse everywhere. And, yeah, it's the nature of stage check and exchange. And over time, the the check 266 00:32:08.190 --> 00:32:08.690 step, 267 00:32:09.150 --> 00:32:10.450 of of that pipeline 268 00:32:11.154 --> 00:32:14.135 will grow more complex over time as we identify 269 00:32:14.515 --> 00:32:22.380 patterns of of data quality issues. Right? So we could we might say, like, hey. If there's a week over week change of more than 30%, 270 00:32:23.080 --> 00:32:31.835 failed the pipeline and send an email to the people, to the owner of the pipeline, for instance. And we have tons of these checks, all all over the place at Airbnb. 271 00:32:32.615 --> 00:32:40.850 So would you say that that's fairly analogous to the idea of unit testing and continuous integration for standard software development but applied to data? 272 00:32:41.470 --> 00:32:49.685 I think it's it's fairly

different, right? Because it's it's reactive and it's production code. So everything, you know, if if you look at continuous integration, 273 00:32:50.225 --> 00:32:51.685 you you're trying to prevent, 274 00:32:52.305 --> 00:33:01.750 logic or changes to make it into into production. So if you do your job well with your unit tests and your continuous integration, you're not gonna deploy 275 00:33:02.050 --> 00:33:04.150 code that that would result in problems. 276 00:33:04.515 --> 00:33:21.325 Where data quality checks and data warehousing might is more of a reactive thing of, like, we have the logic, we don't control, you know, the ingredients that are coming in this recipe, but we'll make sure to, you know, if we detect certain things that we, we cannot detect, you know, we we won't 277 00:33:21.804 --> 00:33:28.544 bring that data into production. So I guess in some ways, you know, instead of being code that would make it or not make it into production, 278 00:33:29.085 --> 00:33:55.615 it would be more like data that would make it or not make into production. So it's it is kinda analogous in some ways. I can I can talk a little bit more about kind of unit testing too and that part of kinda continuous integration for data warehousing? So the state of that is is not really great, and a lot of people ask, like, what is the best practice for airflow? Like, how do you run your unit tests? And how do you make sure that when you launch a pipeline that it's not gonna, you know, break production? And, 279 00:33:56.475 --> 00:34:02.610 you know, the answer is there's probably as many ways to to test and validate a pipeline as there are, 280 00:34:03.010 --> 00:34:17.285 engineers and and pipelines out there. So we have different ways of doing it. But 1 thing that's clear is that you cannot really have, like, an at scale development environment. So you look at when I when I was working at Facebook, we, you know, were close to the exabyte 281 00:34:17.880 --> 00:34:28.285 in the in the data warehouse, and you could not just say, oh, we're just gonna have a dev instance of the warehouse that's also gonna be an exabyte. So really often, you need to create some sort of microcosm 282 00:34:28.745 --> 00:34:49.875 of your, of your data warehouse. And sometimes, they might just be like pushing some very like, some sample data through and making sure that there's no, you know, no fatal errors that are happening and making sure that say all the rows that make made it in the pipeline actually made it or got summarized in the end. The reality is, data engineers don't systematically 283 00:34:50.655 --> 00:34:51.155 test, 284 00:34:51.535 --> 00:35:02.930 a lot of what they do, right? And they they'll probably, you know, test the little piece of pipeline that they work on. So that means they might run that this piece of the pipeline and and diverge it into an alternate 285 00:35:03.325 --> 00:35:12.640 temporary table that they can check. But once, you know, it's

checked, they'll just pipe it right back in and and push it into production. Just because doing things 286 00:35:13.020 --> 00:35:16.480 right or by the book would be super private prohibitive, 287 00:35:16.780 --> 00:35:18.375 in terms of infrastructure cost. 288 00:35:19.075 --> 00:35:25.460 And kind of the how much more confident you're gonna get for the amount of work that that's required, 289 00:35:25.920 --> 00:35:35.195 sometimes it is not worth it. So it's really kind of a case by case, and there's all sorts of techniques out there. I should probably, you know, write a blog post on the subject, 290 00:35:35.655 --> 00:35:42.660 1 day. But the reality is is that data engineers are often less thorough than software developers just because the costs, 291 00:35:43.299 --> 00:35:45.000 involved and the challenges involved. 292 00:35:45.539 --> 00:35:48.119 And have you seen any points where the 293 00:35:48.500 --> 00:35:55.975 work done by data engineers and managers of data infrastructure have bled back into more mainstream software development and systems engineering. 294 00:35:56.675 --> 00:36:06.130 I I know 1 1 aspect of this is that the the same way that data engineers are doing more software engineering, I think it's also true that that software engineers 295 00:36:06.589 --> 00:36:09.035 do do a lot of or a fair amount 296 00:36:09.335 --> 00:36:22.280 of of data oriented work. Right? So in places like Airbnb and Facebook or any modern web companies, a a software engineer needs to be able to set up a an a an AB testing test, right, or an ex an experiment, 297 00:36:22.660 --> 00:36:35.250 so that they can measure whether the change they're making to a website is gonna to measure exactly, like, what how it's gonna influence the different metrics that they're trying to to move. So from that perspective, 298 00:36:35.789 --> 00:36:36.769 software engineers 299 00:36:37.230 --> 00:36:44.145 are doing some of the things that data engineers do. So writing pipeline or instrumenting a metric. Right? 300 00:36:45.485 --> 00:36:56.830 Building dashboards and building their own little, pipeline and dashboard is not uncommon for for a software engineer working working on a product. So how do you see the role of data engineers evolving in the next few years? 301 00:36:57.450 --> 00:37:01.470 I see that that the role is gonna become a little bit more formal and balanced 302 00:37:02.005 --> 00:37:23.615 in regards to data science. So I think a lot of companies have recognized that they needed data scientists in order to compete in in in their field with their, you know, in their business. I I don't think all companies that have identified that they need a data engineer to organize the data structures and pipelines that the data scientists are gonna source from. And 1 fact is 303 00:37:23.915 --> 00:37:34.500 that data data scientists are pretty horrible at building infrastructure and data pipelines. Right? So they'll they'll they'll build these pipelines that are very brittle and will fail over time that are not manageable. 304 00:37:34.800 --> 00:37:40.565 Typically, the data modeling is not done right and there's a lot of throwaway work and redundant 305 00:37:40.865 --> 00:37:48.710 work where all the data scientists will go from the the raw data, right, and then do their own analysis and apply their own transformations, 306 00:37:49.089 --> 00:37:51.795 and then you and then you end up with big problems like, 307 00:37:52.595 --> 00:38:10.795 the the same metric or the metric with the same name that has very different value depending on who computed it, in data science. So companies I think that I've kinda skipped this part of say data warehousing, data engineering, get serious about their data structures and how they organize their data and metadata, 308 00:38:11.095 --> 00:38:17.035 they're suffering because the data scientists are doing that work. They're not that great at it, and there's no consistency 309 00:38:17.494 --> 00:38:36.549 in in the way they look at their numbers. So no 1 you get into this issue where no 1 can trust any number, where the CEO is like, I don't even wanna see the dashboard because I know it's wrong because it's different from that other dashboard. So I see also like data engineers starting to do more abstracted work. Right? So that means whatever a data engineer 310 00:38:36.849 --> 00:39:13.180 does today, which is building pipelines and building the warehouse, I see that engineers starting to build services and frameworks that, that other people within the company can use. So an example of that would be say an an Or an obvious example is the AB testing framework, where maybe, originally, you would have for every experiment, some data scientists would go and do all the stats work necessary to figure out whether that experiment, was successful or not or how it moved different metric. Now the data engineer along with software engineers and you know, with data scientists that together will build the reusable framework that can be used for all experiments. 311 00:39:13.640 --> 00:39:15.980 But that you know, that's just 1 application 312 00:39:16.360 --> 00:39:29.460 of reusable components that data engineers can build. So there's other ideas like core analysis, aggregation framework, you know, all sorts of computation framework that abstract out the common data work that's done 313 00:39:29.920 --> 00:39:31.300 in in different companies. 314 00:39:31.600 --> 00:39:38.635 I see the forces I was talking about earlier, so over the next few years, our data engineers are gonna align 315 00:39:39.095 --> 00:39:42.395 more with their verticals or are they gonna align 316 00:39:42.695 --> 00:39:52.740 closer to their infrastructure peer and work on more horizontal product? So that's still unclear to me and there might be some we have to figure out which way it's gonna go. 317 00:39:53.215 --> 00:39:56.515 My draw and my personal interest is to go towards more

318 00:39:56.895 --> 00:39:57.395 horizontal, 319 00:39:57.935 --> 00:39:59.155 than than vertical, 320 00:39:59.535 --> 00:40:01.230 but I'm not sure which way, 321 00:40:01.630 --> 00:40:11.565 the industry is gonna go. And then 1 question around the the future and the next few years, and 1 thing that I I truly hope we start to see is some sort of convergence 322 00:40:11.865 --> 00:40:38.590 in in the data ecosystem. So that means, you know, can we all agree that you know, we should all use Spark Streaming and not Samsa or can we all agree kinda what happened with Kafka? People kinda converge and say Kafka is the tool that we're gonna use, and you won't have to choose between 5 or 6 different tools or frameworks. So hopefully, there's convergence in that area where, you know, people agree on how they should build it and maintain, their infrastructure. 323 00:40:38.965 --> 00:40:43.545 And also, you know, I'm hoping that there's you know, when you think about the data infrastructure 324 00:40:43.845 --> 00:40:48.750 work which is installing and maintaining, you know, Hadoop clusters and Druid clusters 325 00:40:49.630 --> 00:40:54.690 a Spark streaming cluster. I'm really hoping that some cloud providers or some service providers 326 00:40:54.990 --> 00:41:01.015 commoditize that so that we we don't so that every company does not need to go and and install, 327 00:41:01.634 --> 00:41:05.974 and and maintain that all of that infrastructure. It's just inefficient. 328 00:41:06.820 --> 00:41:11.480 So that might be a trend in the future where we see some really good service providers, 329 00:41:11.860 --> 00:41:17.775 where you can stand up, you know, a a Druid cluster in in 10 minutes and and get value from it right away. 330 00:41:18.395 --> 00:41:23.250 1 of the things that you sort of alluded to briefly is the idea of 331 00:41:23.630 --> 00:41:27.810 data scientists writing some analytical code that then needs to be operationalized. 332 00:41:28.110 --> 00:41:29.650 So how much of that responsibility 333 00:41:29.950 --> 00:41:31.890 falls on data engineering versus 334 00:41:32.234 --> 00:41:38.655 a more full fledged software engineer who needs to then take that analytics code and write it as a scalable piece of software? 335 00:41:39.250 --> 00:41:41.910 Yeah. That's a very good question as to like, whose responsibility 336 00:41:42.290 --> 00:41:44.550 is it to, make the data scientists 337 00:41:44.850 --> 00:41:56.779 obsolete. Right? Is it is it the data science job to, kinda automate their own work? I don't I don't think it's their their natural draw since they're not necessarily engineers and, you know, their skills don't necessarily align 338 00:41:57.160 --> 00:41:58.380 that way. At both, 339 00:41:58.759 --> 00:42:03.000 at both Airbnb and Facebook, which are my 2, most recent company, there is, 340 00:42:03.575 --> 00:42:05.015 some form of a, 341 00:42:05.655 --> 00:42:06.555 machine learning 342 00:42:06.855 --> 00:42:07.355

infrastructure 343 00:42:07.815 --> 00:42:08.635 type team. 344 00:42:09.175 --> 00:42:11.675 I believe at Facebook, it was called data science infrastructure, 345 00:42:12.055 --> 00:42:13.430 and at Airbnb, 346 00:42:13.810 --> 00:42:17.030 I believe it's called, either machine learning infrastructure 347 00:42:17.330 --> 00:42:22.914 or data science infrastructure as well. And that's that's a slightly different role. Are 348 00:42:23.295 --> 00:42:39.135 data engineers the right people to do this? I guess we're getting into 1 of these gray zone where we're looking for a unicorn type of person that has, you know, all those skills at the, you know, in in the middle of that Venn diagram of data science, software engineering, data engineering, 349 00:42:39.595 --> 00:42:41.215 and those people are extremely, 350 00:42:41.835 --> 00:42:44.795 rare. If you find 1, just, make sure they don't, 351 00:42:45.480 --> 00:43:03.690 you keep them for a long time because it's almost impossible to find these people. And it's and it's hard for for people that just so say, if you put 1 data scientist, 1 data engineer, and 1 software engineer in a room and you try to make them work together, you know, the results may vary. But, you know, I I think this idea 352 00:43:03.990 --> 00:43:04.470 of, 353 00:43:04.950 --> 00:43:06.250 data science infrastructure 354 00:43:06.870 --> 00:43:13.194 is super super interesting, probably the most 1 of the most exciting area right now. Right? Where we can commoditize 355 00:43:13.655 --> 00:43:23.970 data science. And I believe in a lot of cases, as I said earlier that, you know, data science value is not necessarily to do, like, 1 really intelligent machine learning 356 00:43:24.510 --> 00:43:33.665 model to solve 1 simple problem. But I I believe the real value is to do some basic, you know, machine learning across the board. Right? To not necessarily do something 357 00:43:33.965 --> 00:43:38.630 very complex in terms of of like the math or the libraries 358 00:43:39.010 --> 00:43:39.990 or, you know, 359 00:43:40.370 --> 00:43:41.750 doing some very complex 360 00:43:42.050 --> 00:43:45.250 data science work. And really often it's just about having some some, 361 00:43:45.665 --> 00:43:49.845 just a little bit of machine learning applied in the right place and across the board then commoditizing 362 00:43:50.145 --> 00:43:54.484 that. So hopefully we're gonna we're gonna see some of that, like right now I think it's only, 363 00:43:54.799 --> 00:43:56.180 you know, a handful of companies 364 00:43:56.480 --> 00:44:06.165 doing that and and then kinda struggling. There's not really companies that are that are doing that or services or libraries that are provided to do that sort of stuff. 365 00:44:06.785 --> 00:44:10.645 So are there any other topics that you think that we should cover before we close out the show? 366 00:44:11.000 --> 00:44:11.900 No. I think we covered 367 00:44:12.599 --> 00:44:13.420 a lot of things. 368 00:44:14.040 --> 00:44:16.619 So, yeah, I think I think we we're pretty good. 369 00:44:16.920 --> 00:44:30.670 Yeah. It's definitely a large space with a lot of different concerns, but I think that it's important to sort of lay the groundwork of when you say data engineer, what is it that you're really talking about? And I think that we did a pretty good job of at least doing a good approximation of 370 00:44:31.290 --> 00:44:49.160 that. Right. At least like centering, find finding the middle point of, you know, the core of what is a data engineer, and then from that point, you decide how far from that point you wanna allow people or people to go to. Alright? But the core to me is right around, you know, the data warehouse and and the pipelines that organize 371 00:44:49.460 --> 00:44:50.119 the company's, 372 00:44:50.420 --> 00:44:51.480 data and metadata. 373 00:44:52.180 --> 00:45:12.205 Alright. Well, for anybody who wants to keep in touch with you or follow what you're up to, I'll add your preferred contact information to the show notes. I just wanna say I really appreciate you taking the time out of your day to share your thoughts about data engineering and your experiences working with it. And I hope you enjoy the rest of your evening. Thank you very much. That was an honor, and a pleasure to be on the show.