TRANSKRIPT 🔗
WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 11/18/2024 00:28:58
Duration: 2388.485
Channels: 1
1
00:00:13.394 --> 00:00:26.890
Imagine catching data issues before they snowball into bigger problems. That's what DataFold's new monitors do. With automatic monitoring for cross database data diffs, schema changes, key metrics, and custom data tests, you can catch discrepancies
2
00:00:27.190 --> 00:00:29.610
and anomalies in real time right at the source.
3
00:00:30.150 --> 00:00:38.890
Whether it's maintaining data integrity or preventing costly mistakes, DataFold monitors give you the visibility and control you need to keep your entire data stack running smoothly.
4
00:00:39.575 --> 00:00:41.675
Want to stop issues before they hit production?
5
00:00:41.975 --> 00:00:43.915
Learn more at data engineering podcast.com/datafold
6
00:00:45.335 --> 00:01:00.399
today. Your host is Tobias Macy, and today I'm interviewing Ken Pickering about streaming data into a Trino and Iceberg lakehouse. So, Ken, can you start by introducing yourself? Hi. Yeah. Thanks for having me on. I, I am the VP of engineering at Going, which is a company that provides,
7
00:01:00.780 --> 00:01:18.325
travel deals for consumers. And do you remember how you first got started working in data? It's a long time ago. So I came up from, like, product engineering, but I found over time that more and more of your the requirements I was working on involve data in some capacity. And so I think it it started with, ecommerce
8
00:01:18.865 --> 00:01:32.690
and InsurTech, really, which were based on pretty large volumes of data at the time. So recommendation algorithms for ecommerce. And then, like, I worked for a property insurance company where we'd, like, be able to price, like, roughly what would be in a person's home, you know, based on region,
9
00:01:33.150 --> 00:01:35.325
size of house, like, that sort of thing.
10
00:01:35.725 --> 00:01:41.585
And so that's where I really got interested in the topic of it. And and more and more of my career has, like, pushed further into,
11
00:01:42.365 --> 00:01:43.825
interpreting and then productizing
12
00:01:44.125 --> 00:01:45.585
data in some capacity.
13
00:01:46.125 --> 00:01:47.825
Your current role is
14
00:01:48.390 --> 00:01:51.690
in Going where you're leading their data platform,
15
00:01:52.070 --> 00:02:00.570
and I'm wondering if you could just start by giving a bit of an overview about the types of data that Going relies on and the role that it plays in the business.
16
00:02:01.034 --> 00:02:16.015
So yeah. So, Going needs to find travel deals. It's a subscription service. So you sign up, and we try to find travel deals for you. And more and more that becomes a data intensive problem. And there are sort of two aspects of that. 1 is, well, how do you find flight prices? You know, it's a
17
00:02:16.689 --> 00:02:33.165
very large problem domain. You know? Instead of computer science terms, it it could be classified
Load full transcript
in some capacities as, like, NP complete. You know? Because it you can get to a lot of places via a lot of routes. And so, like, trying to find deals for customers is is sort of part of that that problem space. And, you know, prohibitively,
18
00:02:33.545 --> 00:02:50.450
the volumes of data I mean, the data the data volumes you're dealing with are are prohibitively large in some capacity, and so you have to figure out how how you wanna traverse that. And so, you know, ingesting large volumes of of flight information, and flight data is one is one aspect of it and building models around. Well, is this a good price or a bad price?
19
00:02:50.750 --> 00:03:05.295
You know, if you have a specific itinerary you're trying to follow that's different than finding a globally available, like, deal. And so, like, we try to focus on sort of both those problems and just advise people when a good time of travel would be. You know, and then then there are concepts like, you know, personalization.
20
00:03:05.755 --> 00:03:11.470
Like, what are the person's interest? What kind of deals apply to them? Because when you deal with, like, a mobile form factor,
21
00:03:11.850 --> 00:03:25.069
making sure that you're giving, like, timely topical and and sorted information is interesting. And so and I'd say expected from a modern consumer as to how they use their apps today. So we you know, and so I'd say we collect, yeah, basic travel information,
22
00:03:25.370 --> 00:03:43.690
low flights cost, you know, segments and routes, and all that sort of fun stuff. And then the other side is, like, event stream information. What are people doing on the device, and how can we provide a better service to them based on based on that input? You mentioned a little bit about the size of data that you're dealing with. Yep. But in terms of the overall volume variety velocity,
23
00:03:44.150 --> 00:03:48.650
what are some of the characteristics that you need to accommodate to be able to
24
00:03:48.950 --> 00:03:52.810
fulfill the data oriented objectives of the going system?
25
00:03:53.315 --> 00:04:23.884
So one way we've solved the problem is is both through passive and active search. And passive search is where the lot a lot of data comes in. Active search is when we search for flight prices. It's a transactional where, you know, we send out a query. We get a response to request coming back in, and it's a pretty manageable data. The passive data is sort of like where we, aggregate with a very large it's called a global distribution system, GDS. It's how a lot of online travel agencies search for airfare, and what we do is we get access to their global traffic. And so,
26
00:04:24.345 --> 00:04:29.405
each of them can give you about, at least a gigabyte a second of of travel data.
27
00:04:29.865 --> 00:04:59.044
And, you know, if you do the math out, it's about 50 petabytes a year of data you're kinda sifting through for each provider. And so, like, that is sort of the relative volume of it, and it's kind of messy data and that it's, like, really, like, nested JSON. And so, you know, there is a fair amount of preprocessing you have to do to be able to derive action from it. But our goal is to try to kind of sift through that haystack and find a needle of, like, good deal flight pricing for our customers. And so, you know, I think the the reason that we solve the problem with streaming in many capacities is because,
28
00:04:59.505 --> 00:05:16.510
time matters in these types of situations because flight prices fluctuate. The sooner we identify a deal and the sooner we notify our consumers of a deal, the more likely that deal is being able to exist when they go to transact on business. Because, you know, by prices can change a few times a day. And because you don't know
29
00:05:16.889 --> 00:05:23.705
how long that deal has been in place in some capacity, you get a notification of it from a passive search that somebody else did, but you don't really know
30
00:05:24.005 --> 00:05:46.319
how long that price has been active because there's not really a great system for publication and broadcasting that. So that's why we went went with a kind of a real time streaming nature of the architecture because of the the cost considerations. That's also why we tried to land it in our warehouse and and and iceberg in real time. Because if you're already processing the data because it's, like, a large amount of data and it's kind of a messy format and you already have extracted
31
00:05:46.699 --> 00:05:49.280
what you're looking for out of it, it makes sense to transform
32
00:05:49.685 --> 00:05:53.225
and land as well. And before we get too much more into
33
00:05:54.005 --> 00:05:55.145
the technical architecture,
34
00:05:55.605 --> 00:06:04.345
I'm also interested in the composition of your engineering and data teams that manage and support and consume these data flows where,
35
00:06:04.770 --> 00:06:08.950
typically, if you're dealing with analytical data, then a lot of that responsibility
36
00:06:09.570 --> 00:06:16.230
lives with a separate team from your software and application engineers. But given that the data is
37
00:06:16.545 --> 00:06:18.005
a core capability
38
00:06:18.385 --> 00:06:26.405
of the actual product, I'm curious how that influences the ways that you think about the composition of that engineering talent. Going engineering is small,
39
00:06:26.785 --> 00:06:41.160
small small and fierce. We're it's about 25 people right now. So we all work together. I mean, we're we're still a startup in that capacity where there aren't really huge organizations and big departmental silos. But, yeah, it does require close coordination
40
00:06:41.539 --> 00:07:44.544
between how we are operating as as a team. So we do have people that specialize in data engineering. We have 2 data engineers and a manager, that oversees the 2 data engineers and our one data scientist, Nick, who does an amazing job for being
our solo data scientist. But we really have a lot of vendor tooling. I I mean, in this ecosystem, like, when you don't have very many engineers, we've kind of strung together a series of SaaS tools that let us streamline engineering operations. So we're mostly focused on, I would say, how we efficiently transform data on the data engineering side and then how we efficiently model that data on the data science side and worry less about, like, the operational side of data engineering, which we, like, we push out to vendors. But and then in terms of how how people interact, like, a lot of it comes from product management. Like, we will develop models, at the end of the day, you know, we have to work closely with product and the product teams be like, okay. We have a statistical valuation of this price, but how do we present that to a consumer in a way that makes sense? How do we measure, like, notification threshold and, you know, when we should push something to users versus when we should observe and see if a price gets better and then talk to our customers about how do we display
41
00:07:45.005 --> 00:08:01.819
a lot of the rich information we collect and generate to customers so they can feel like they're making an informed and educated decision on a flight purchase. Being a subscription business, we sort of have to do that because that's our credibility. We don't make money from the transaction. We make money on advising people of a potential transaction.
42
00:08:02.280 --> 00:08:29.600
And so accuracy and and empowering our customers is actually very important for our business. And I don't see a better way to do that than, like, close partnership with data and data science. You know, I mean, you know, we collect tons and tons of information on their behalf and then try to represent that in a way that is that is timely and makes sense, but our credibility is also in those numbers. And so people will unsubscribe if they don't see value in in the company. And so that's really one of the major things and and and why we really require close-up participation
43
00:08:29.979 --> 00:08:36.160
among the teams. Now digging more into the architecture, you've already mentioned Iceberg. We already have mentioned Trino
44
00:08:36.565 --> 00:08:43.625
at the start. You've stated that you're dealing with streaming data flows. I'm wondering if you can just talk to some of the ways that that overall
45
00:08:43.925 --> 00:08:50.105
system design fits together and the ways that you're accommodating streaming in that overall architecture.
46
00:08:50.510 --> 00:08:55.089
So we're heavy users of Confluent to collect the streams. We use, Starburst Galaxy
47
00:08:55.390 --> 00:08:56.210
to tie into
48
00:08:56.750 --> 00:08:59.089
Confluent and then transform to Iceberg.
49
00:08:59.390 --> 00:09:05.010
There are some intermediary programmatic steps that occur also derived off of Kafka with the separate services.
50
00:09:05.365 --> 00:09:38.675
And then on the
modeling side, we use Databricks. And so Databricks ties to our iceberg data warehouse to compute models, and then we have services that run-in real time that can latch on to Confluent to parse events in real time or a real model. When you say model, I'm wondering if you can just unpack that a little bit because it is a very overloaded term. There's a variety of different applications we use. A a lot of it is just predictive analytics. You know, like, if something is a good deal or not is, you know, we we use, like, z clustering for that in terms of, like, okay. Like, on what threshold of a deal does this sit? And then, you know, we are getting a little more into
51
00:09:39.135 --> 00:09:44.595
ML because we have to when you when you start incorporating things like advanced notice and seasonality
52
00:09:45.055 --> 00:10:02.825
and velocity of, like, number of searches on a domain, like, the problem space gets progressively more complex if a customer says, like, I wanna fly specifically on this weekend. Right? You can analyze a whole year's worth of data and determine, like, what the highs and lows of pricing are for that year. But, like, to find the right price for the right duration and a right timeline,
53
00:10:03.385 --> 00:10:21.480
incorporates progressively more complex techniques to do that. So yeah. Like that but that's how we're doing it today. And then there's, like, there is the BI side of how we model and cluster user behavior and cohort and and those sorts of things. Now we've established what your current architecture is, some of the different system components. I'm wondering
54
00:10:21.860 --> 00:10:29.160
whether that is how the overall system design started or whether you had maybe a more batch oriented
55
00:10:29.714 --> 00:10:38.295
approach to begin with and maybe what the evolution looked like and some of the challenges of managing that migration process while keeping the business operational?
56
00:10:38.834 --> 00:10:45.175
No. When I started it's been about a year. It's been after about a year. It was really right centered around a a a Ruby monolith
57
00:10:45.530 --> 00:10:58.750
and then a tool that would ingest data but not capture data, and and it was kind of like an expert rule system as to how we found flight pricing. But kind of rudimentary and very rule specific and focused, and the rules were kind of tweaked by by humans.
58
00:10:59.065 --> 00:11:24.860
And so one of the things I wanted to do when I came in is it's a it's a you're not gonna actually be able to scale with a system like that. The order of magnitude of where we would need to go and still be human curated is just it's, you know, it's just not possible, honestly. And so so, you know, we were a Snowflake customer at the time, a lot of batch oriented processes in Snowflake. I chose to go to a more open stack just because I saw the volumes of data, we would be ingesting
59
00:11:25.185 --> 00:11:39.445
as a business and thought the cost magnitude
in Snowflake would be eventually prohibitive. And so one of the reasons I like the open lake house architecture is that it lets me commoditize compute and lets me use different vendors for different purposes and actually really tweak and refine,
60
00:11:39.750 --> 00:12:31.135
how we handle things. Like, in house today, we use Spark for a couple different things. We use Athena. We were using Athena to load into Databricks initially because it was the easiest way to get data to Databricks. You know, now Databricks, though, has acquired Tabular, and now they have a native Iceberg integration. And so we're talking to them about, like, maybe potentially using that. So it's but, like but those those options are open to us because we follow an open lake house architecture. So that was really an important thing for me to lay out just because I wanted the optionality for the business to be able to expand in a bunch of different directions. And I think if you're building a data data derived company today, I do think it's a better pattern. I think being able to separate storage and compute with a format that is both fast and has a variety of, I'd say, cool functions, in it is is probably what the future should be. And so that's sort of where I netted out on that. And so that was the first thing we laid out. But now now that we have, like, unified ways to process data and handle pipelines,
61
00:12:31.515 --> 00:12:38.209
it's it's a lot easier to incorporate new data sources, to incorporate new models. One of the things I'm working on is, like, data democratization
62
00:12:38.670 --> 00:13:15.509
for our business customers and how we actually get out of having analysts have to author every dashboard and and data engineers have to do every aggregation under the sun. To try to make data the center of the business is really kind of my major focus. Because if you think about what we provide, we're the recommendation engine for flight pricing, which is all data. It is all it is all computation. It is all data. We'll have a UX. We have, you know, brand specifications. We have marketing and efforts and those sorts of things, but those are also derived from data and our user behavior. In terms of the selection of systems and the structure of the overall architecture, I'm also interested in maybe the the whys and wherefores
63
00:13:15.889 --> 00:13:20.565
of how you ended up with Iceberg and Trino as the core of that system,
64
00:13:21.025 --> 00:13:29.125
particularly given the streaming nature and the fact that you are using that data for a consumer facing
65
00:13:29.585 --> 00:13:33.320
use case. And Trino is definitely very scalable
66
00:13:33.780 --> 00:13:45.000
and performant, but not typically in the way that you would think about from an application design perspective where you want to have query responses in the order of milliseconds or maybe
67
00:13:45.615 --> 00:13:47.555
low 100 of milliseconds at the most.
68
00:13:48.255 --> 00:14:09.130
And,
also, on the iceberg side, typically, when you think streaming lakehouse, I know that Hoody has positioned itself as the answer to that because of that was the core of what it was designed for. And I'm wondering kind of what that evaluation and selection process looked like to get to where you are. Well, choosing format was an important distinction for us because of the volume of the dataset.
69
00:14:09.430 --> 00:14:10.970
It gets primitively challenging
70
00:14:11.350 --> 00:14:12.410
to switch formats.
71
00:14:12.865 --> 00:14:30.100
And so when we did the initial analysis of popularity, the formats, what what we see vendor integrations because as I sort of mentioned earlier, we are vendor centric. Where I see the market going, you know, where I see the most effort businesses are putting into the format, Iceberg kinda stood out for us. I'm familiar with Hoodie and and and its capabilities,
72
00:14:30.480 --> 00:14:41.380
you know, and I'm familiar with Delta and its capabilities. But for me, it was like a VHS Betamax thing where you don't really wanna be putting everything in Betamax. And so for me, it was targeting
73
00:14:41.840 --> 00:15:26.160
where I thought the business was going. And, you know, it turned out I think I think I'm still correct. I think it turned out to be a correct decision for us. You know, I I like Trino for analytic purposes and, purpose of aggregation huge datasets. Like, that's what we mostly use it for today. For consumer data, I am all for caching and or or pushing data to different data sources. For instance, we heavily use, like, a open search for our site for, like, deal searching and those sorts of things. You know? And so for me, it's like, if I have to lift and shift for consumer applications and cache, like, current stuff outside of our warehouse, we'll do that. You know? I think it's it's really, like, right tool for the right job. And so, you know, Trino is is an analytics beast and and, you know, I think and it's been great at processing these large datasets. I I probably wouldn't use it for right right to consumer applications
74
00:15:26.540 --> 00:15:56.640
unless I thought it was more cost performant than an Elastic. But the volumes I'm pushing into Elastic are not prohibitive enough or the index is too expensive yet. And so that's where we are today. But I think one of the best parts of an open ecosystem is having multiple engines be able to tie into it and having multiple patterns with which to extract data. And so that's really what we've targeted and focused on. But I sort of have to I do have to abstract what is required to operate a real time app where people expect fast results versus, like, being able to seek across large volumes of data for for calculation
75
00:15:57.260 --> 00:16:05.675
purposes and then eventing purposes. Because, if you think about the way a lot of people engage with apps, this is the way we try to encourage people to engage with apps, is it comes down to push notifications.
76
00:16:06.055 --> 00:16:08.075
So, like, what drives our push is,
77
00:16:08.775 --> 00:16:38.105
you know, massive volumes of data and massive volumes of processing, and then we decide when to push to our customers. And that is basically based on our historic and and data warehouse data. When we do a push, though, we want someone to come into the app and have a fast response. And so, like, the things we have notified them about will be fast and current for them so they can just pop in, look at the deal, determine if they want it, and take an action. You know? And so you can kind of work both sides of that and try to find the right technology mix, to do that sort of thing. And I'm also
78
00:16:38.485 --> 00:16:40.745
wondering if you have done any evaluation
79
00:16:41.445 --> 00:16:45.785
of some of the recent OLAP engines. I'm thinking in particular,
80
00:16:46.860 --> 00:16:52.880
in terms of ClickHouse as one of the popular options. Although the the fact that it has the collocated
81
00:16:53.260 --> 00:17:10.705
storage and compute, I can see as being a potential reason against it in your case. But, also, the PINO project is another one that comes to mind as far as an option for OLAP and for being able to actually directly serve some of those end users because of the fact that it is optimized for those fast query responses.
82
00:17:11.085 --> 00:17:18.170
I play with both Pinot and ClickHouse, I think I think, we're like, I don't wanna say anymore about, like, kind of reservability system that I'm 99%
83
00:17:18.550 --> 00:17:41.580
sure it uses ClickHouse based on the cost economics of it. For me, it's in engineering leadership, sometimes I just we pick something and we stick with it until it prevent to cause us problems later. And so I have a lot of intellectual curiosity and Pinot and ClickHouse and those sorts of things. Like, honestly, Postgres and Elastic are doing the consumer job for us today. And until it becomes, like, cost prohibitive or scale prohibitive, we probably wouldn't take a step back and reevaluate
84
00:17:41.880 --> 00:17:54.220
because it is about cost performance. And until something becomes a problem there, like, I wanna keep engine focused on more of, like, the intellectual property generation than than kind of cutting over data stacks. And so if performance became a problem, we would a 100%
85
00:17:54.600 --> 00:18:59.045
evaluate that. Right? It's just it's just not a problem today. And so I think it's just, you know, setting up your KPIs effectively to say, alright. The app has to respond in this amount of time. If it can't I mean, that that's why that's why Elastic was brought in in the first place. Right? Was that relational database is way too slow in assembling, you know, assembling results for some of these queries. It was just a lot faster to get the results out of out of Elasticsearch. I know Elastic has limits. I've used Elastic extensively in my career, and at some point, we
would look at, like, yes. Like, we will probably end up needing to replace that at scale. But for today, you know, with kind of, like, awkward teenage years as a business, it's also cost effective and appropriate. So And I imagine too that part of the ways that you are transforming and representing the data as it gets to that consumer interaction layer also makes it more conducive to a record oriented approach where Postgres and Elastic are able to operate across it effectively. You're not going to have a pattern where the user interacts with the application and says, I want to search for a flight from Boston to Berlin, for example, and then you then have to go and scan across 1,000 or millions of rows. You've already precomputed
86
00:18:59.425 --> 00:19:00.965
the optimal options
87
00:19:01.265 --> 00:19:03.125
for that pairing of locations.
88
00:19:03.750 --> 00:19:14.090
Especially with an app form factor, you really have to craft your your the data you're returning to the app to the form factor because you don't wanna send apps tons of data. They don't have great processors.
89
00:19:14.470 --> 00:19:17.290
They could be unlimited bandwidth. And so, extraneous data tends to slow down your customer experience. And so the
90
00:19:18.945 --> 00:19:19.445
experience.
91
00:19:20.304 --> 00:19:22.085
And so the the hope would be that you are notifying them and targeting something specific and exactly sending
92
00:19:22.465 --> 00:19:23.684
them that data. And if they wanna do expiration, that's
93
00:19:24.065 --> 00:19:24.565
fine.
94
00:19:25.105 --> 00:19:27.044
But the the good thing is,
95
00:19:31.240 --> 00:20:06.880
in travel, people expect the experience to be slow. So once people get off the well worn path of, like, here's a notification. Here's what we found for you. You know, it's a fast fast experience there. People have kind of learned that, like, if you search for a flight, you're gonna sit there for 30 seconds. You know? And and you we've all used Kayak or Expedia, and you see the flight results. Yeah. I mean, it's you you can make it look interesting, but at the end of the day, those systems are all slow. And so I get I get a little bit of a pass on speed just based on people's normal experience with travel. But, you know, at at the end of the day too, though, the problem is you're also getting, in many cases, hundreds of results back from multiple
96
00:20:07.340 --> 00:20:15.925
providers that you have to mesh together and then, return to the customer. And so that's actually also where we go back and forth between REST and and gRPC.
97
00:20:16.305 --> 00:20:25.685
Right? Like, when do you stream data versus when do you, have a RESTful interaction? Because, like, you know, in some some cases, they're definitely streaming cases. In terms of the orchestration
98
00:20:26.225 --> 00:20:28.885
of the overall end to end data flow,
99
00:20:29.400 --> 00:20:31.020
Obviously, you have your
100
00:20:31.320 --> 00:20:31.820
Kafka
101
00:20:32.200 --> 00:20:42.860
stream for the data ingest. You have Trino and Iceberg for manipulation and aggregation. You've got Postgres and Elastic for presentation. But how do you manage the
102
00:20:43.245 --> 00:20:43.745
triggers,
103
00:20:44.045 --> 00:20:46.625
the workflows, the dependencies, and observability
104
00:20:47.005 --> 00:20:55.345
across those different data streams to ensure that you're achieving the latencies that you're targeting and making sure that any errors are
105
00:20:55.645 --> 00:20:56.145
identified
106
00:20:56.845 --> 00:21:06.289
ahead of time? And this is probably a separate question. But given the fact that these are consumer facing data flows, I'm also interested in what the uptime
107
00:21:06.669 --> 00:21:07.169
requirements
108
00:21:07.470 --> 00:21:11.650
are and some of the on call requirements that you have around those data flows.
109
00:21:12.005 --> 00:21:17.785
Yeah. Right now, we are a big Datadog shop, and and I find companies our face typically use
110
00:21:18.245 --> 00:21:31.970
Datadog, I would say, in general. Like, companies our size use Datadog just because it is, like, an all in one shop for all forms of observability and lets you stand up things rapidly with with with minimal effort from engineering. You know, I think as engineering shops get more,
111
00:21:32.590 --> 00:21:46.155
more advanced, like, you start leaning more of the Prometheus route or or looking at other systems that can do this. So tracing and analysis just because data gets expensive for a lot of this stuff. And but right now, that's what we're we're mostly targeting. I would say
112
00:21:46.615 --> 00:22:18.025
in terms of, like, uptime and on call, like, instant response is a thing. You know, and we try to categorize different forms of instance. Like, if if the app or site is down, it's definitely like a p zero all hands on deck. Like, you know, get we gotta get it back up and running. Depending on what pipelines fall or or, you know, we can we can kind of order those based on order of magnitude. We try to have runbooks and parties in in each of the alerts. So we know something's going on. We can alert on it and then decide what the relevant action is. We have a blue team internally that responds to events. Engineers take a rotation where there's a primary and secondary.
113
00:22:18.485 --> 00:22:29.640
Now, your secondary for the 1st week, primary for the 2nd week is kind of a hand off and transition of shadowing. And since and then, and that's worked for us. I'll say, like, we we target 99.9
114
00:22:30.180 --> 00:23:00.169
because that well, because also, that's what most of our vendors give us, and so it's really tough for us to stay up past 99.9 if it's the only certainty we're getting from other places. But I would say that, like, it's not we're we're also not like a hospital. You know? And we're so there there is some amount of tolerance for downtime because it's not, like, a critical operating
system like I've I've had to work on in previous aspects. I feel we're talking, like, 999 or those types of those types of systems. But we are a subscription service, and it's frustrating for our customers when we are down. And so we try very hard to hit that 999.
115
00:23:00.549 --> 00:23:01.610
As far as the architecture,
116
00:23:02.149 --> 00:23:36.285
I imagine that because of the fact that you are using Postgres and Elastic on the serving and presentation layer, you're caching the data there. You have a certain amount of room for errors earlier in the stack because you're not going to bring down the whole site if you fail to get the latest batch of data from one of the data providers. It's just going to be stale for a little bit longer than expected. And I'm wondering if you can talk to maybe some of the ways that you are managing the overall data life cycle because, obviously, there is a certain
117
00:23:36.665 --> 00:23:37.645
timeline to
118
00:23:38.105 --> 00:24:05.215
the utility of flight data where once the flight date has passed, your customers don't care anymore. It's useful from a analytical perspective to be able to maybe do some projections and analysis of whether a particular price is good or not. But for the consumer, they don't care once that flight has flown. And I'm wondering if you could talk to some of the ways that you manage the life cycling of data in that presentation layer, how far into the future it can even project
119
00:24:05.674 --> 00:24:22.640
versus the data size and, performance requirements on the consumer layer. Yeah. And that's where the active and passive shopping comes into the mix because passive data for me is opportunistic data. And that, like, we don't pay it it it it's much cheaper for us to look at other people's searches than to execute our own searches based on the way search query and and performance works.
120
00:24:22.985 --> 00:24:28.925
Execute our own searches based on the way search query and and performance works. So at the end of the day, I save bottom line money by processing huge amounts of data. But at the same time,
121
00:24:29.305 --> 00:24:33.805
like, if if I don't get, for some reason, any searches
122
00:24:36.690 --> 00:25:51.695
from, like, you know, JFK to Hawaii, you know, because a lot of people wanna go to Hawaii, then I have to active search for that data, which means we'll pay for it and do a ton of searches to try to collect pricing information. And so, like, that's really where the differentiation is. It's like, how are you populating your flight? I I mean, I should have called it like a flight cache. Like, update like, every day we try to get results from passive data to know what existing flight prices are for a huge amount of time line. That's why the data volumes are so pretty large in in many cases is because we are getting a lot of that. But I have to layer on top of that and check the cache if I haven't gotten events and then actively search for and populate that data. Because if someone is watching a flight, they're paying us to watch it for them, and so we have to find that data if we don't have it, in our system. So that's sort of how how that that side of it works. You know? And so for me, it's if a feed goes down, like, one of the large feeds go down and I miss an hour or two's worth of data, I miss potential deals, and I miss cost savings based on the data that I could populate, but I can still find that data because I can still search for that data. And so but that's you know? But doing this in a way where we can remain cost competitive, it requires passive data to do it because it's just oh, I would pay 1,000,000 and 1,000,000 and 1,000,000 of dollars every month for the number of searches we need to do. And for the orchestration
123
00:25:52.635 --> 00:25:56.970
piece of ensuring that the propagation of data is
124
00:25:57.289 --> 00:26:15.795
proceeding, what are you relying on for that overarching control plane? That's the thing we're still working on. And so right now, we will run a job scheduler to search because we have a a number of users who have set up stuff in the system, but it's still a finite number that we can process. And so we can scan that, check the cache if it doesn't exist,
125
00:26:16.335 --> 00:26:42.875
repopulate from active search. At some point, though, we'll probably need a control point mechanism that will auto populate the cache and and do a lot more intelligence. But for now, we're kind of hacking it with just what we know what people are looking for. You know, again, but that number gets prohibitively large, and we have to think of a new system to build as a result of that. As you continue to build and iterate the system, you mentioned that it's been about a year in the making to where you are now from where you were previously.
126
00:26:43.975 --> 00:26:46.794
What are some of the other technologies
127
00:26:47.254 --> 00:26:48.794
or other architectural
128
00:26:49.095 --> 00:26:51.570
components that you are currently
129
00:26:52.110 --> 00:26:58.130
evaluating, planning for, and what are the pieces that you feel are fairly well established
130
00:26:58.430 --> 00:27:21.549
and are not at any risk of changing? Well, I mean, it's funny because it's like you can swap in components. Like, there's some of this different componentry for some of these things. Like, fundamentally, I do believe in event streaming. Like, I believe in, like, kind of a a message bus style or event streaming architecture. There are different technologies in there. We use Confluence today. We could use insert service there tomorrow. Right? There's competitors to to Confluent and Kafka now. You know, it's like,
131
00:27:22.270 --> 00:27:53.800
and so for me, it's about what makes sense in the natural evolution. I would say, like, the one thing that is sort of immutable right now for us is iceberg in that we are landing in it and we have no desire to leave it. I think,
hopefully, the technology ecosystem mirrors that, you know, but it's Galaxy today, and I really enjoy using Galaxy as a Trino provider. But I could run Trino. I could run Athena. I could run could run Remio on top of Iceberg. You know? And so for me, it's like you know, I don't I'm not looking to switch components out because, well, one, it's sort of a new space, and we just stood it up. Like, it's we we made some pretty pretty self conscious decisions
132
00:27:54.180 --> 00:27:58.680
initially when we selected this stack. Like, you know, we we evaluated SageMaker and Databricks,
133
00:27:59.060 --> 00:28:17.640
on the ML side, and we we went with Databricks. And the odds of Databricks disappointing us for our ML problems in the long term are are low. You know, it's a pretty low I think what we're doing, I think it's a it's a good tool set for that. An area that we're we're we're leaning into in terms of future expansion, though, is LLM. Like, I think trying to figure out how we do content generation
134
00:28:17.940 --> 00:28:18.760
and copy
135
00:28:19.300 --> 00:28:20.760
or or, like or or explaining
136
00:28:21.540 --> 00:28:38.835
our logic or reasoning for why we'd recommend a destination to a customer, but not having to have a human being being write that to them. Because if we know somebody is watching a lot of travel travel places, right, like a lot of, like, tropical travel places, like, I'm looking at Aruba and Hawaii and, you know, let's say, a couple of the Caribbean Islands. Like,
137
00:28:39.215 --> 00:28:41.315
why if we find a deal for,
138
00:28:41.855 --> 00:28:42.595
the Azores.
139
00:28:42.975 --> 00:28:53.870
Right? Like, we have to tell them why we're saying, well, hey. I know you set up these things, but here's why I recommend Azores. Here is what you can do there. Right? Like, well, that's that's a perfect use case for LM. We we've written travel publications
140
00:28:54.250 --> 00:28:56.510
internally in a lot of these places and could probably
141
00:28:56.890 --> 00:29:32.765
summarize that content pretty readily. And so, like, that's an area of, like, let's go over all of our unstructured data we've written over the past 9 years as to how we could, like, reflect that to customers and reflect that experience then is something we're considering. But that would be you know? And I think that that's, you know, we're we're we're talking to some folks about a pilot there, in the first half of next year. Because I think right now, we're we're kind of done with the overall, like, like, data flows that we need in our system. And there it's because it's a new system, it's it's pretty good for now. And so it's like, okay. Now how we can leverage this more? Probably 2 years from now, I'll be like, oh, crap. Like, we should probably reevaluate how we how we do all this stuff. But for now, I feel I feel I feel pretty confident.
142
00:29:33.385 --> 00:29:35.325
Yeah. And to your point, Iceberg
143
00:29:35.625 --> 00:29:39.725
is definitely on the upswing as far as adoption and integration
144
00:29:40.265 --> 00:29:45.030
across the stack. And in particular, I think the Arrow ecosystem
145
00:29:45.330 --> 00:29:51.590
is really investing in support for Iceberg, and so that also expands the interoperability
146
00:29:51.890 --> 00:29:56.950
and the the capabilities, particularly in maybe that more ML oriented ecosystem
147
00:29:57.495 --> 00:30:03.675
because of the investment that's happening in the Python and Rust communities to be able to interop with Iceberg. Yeah. A 100%.
148
00:30:04.215 --> 00:30:18.370
And in your work at going, building this stack, powering the application, what are some of the most interesting or innovative or unexpected ways that you've seen the data or the overall data stack used? So one of the things I love most about
149
00:30:18.670 --> 00:30:22.130
consumer facing businesses is it's all data and insights.
150
00:30:22.510 --> 00:30:27.010
Right? It's like like, it's it's you push a feature, you see how people respond to it,
151
00:30:27.445 --> 00:30:49.539
you respond effective and appropriately. And I'd say, like, learning things about our customers and learning things about behavior and learning things about the changes of technology are always really intriguing to me. Like, minor tweaks in an onboarding flow can wildly unlock different scenarios. And so for me, it's like, well, you don't know that unless you're collecting the data and and getting fine grained about customer behavior in certain instances.
152
00:30:50.000 --> 00:31:55.865
And so I love the insights that that generates. I love the complexity of how we come up with an eventing schema and how we track an eventing schema and how we take millions of events and try to aggregate trends out of them to help drive our business and business strategy. We are a very data centric and data focused business these days. And so I'd say that's it's seeing a feature that you you thought was gonna be minor while they land and seeing the data come in for that, or it's like seeing something that you thought was gonna be awesome, just, like, 2% of users adopt it. And you're like, oh, that's kinda sad. You know? But But those types of things I think are are really intriguing for me. You know, I'd say the other thing I love looking into is global flight trend data. Like, where are people looking to go? Right? Like, what is a desirable location? Like, you know, you could probably see the impact of Taylor Swift's tour across the US in terms of flight searches. It's things like that where you could do weird kinds of relationships. Because, like, you know, for us, like, we're looking into eventing. Right? Like, maybe we should help people fly to see Taylor Swift as a feature. Right? And so we can look at data for that and see if it's a global trend and see what it does with flight pricing and see if it's a thing we could offer our customers. Things like that are pretty pretty neat.
Given the fact that the overarching
153
00:31:56.485 --> 00:32:02.585
mission of the business is helping people get from here to there at a reduced price,
154
00:32:02.940 --> 00:32:30.835
what are the opportunities that you see for incorporating other modes of travel and price analysis on that? I'd love to. I mean, I think, like, I think, you know, European travel train's huge thing. The, and we're starting to look into further partnerships around that. Like, how do we actually help people do, you know, next phase trip planning? Like, flights is part of it. But, yeah, I'd I'd say Goings mission in the long term is to try to be a a generalized travel adviser. Like, we started out with flights, but, hopefully, we could help with accommodations
155
00:32:31.135 --> 00:32:31.875
or recommendations
156
00:32:32.255 --> 00:33:08.750
of things to do or, you know, or offer alternative modes of transportation, let people trip plan, you know, let people share their trips, see what other people are doing around trip planning. I think those types of things are are kind of interesting for us to lean into. But, you know, it's like, you know, 25 engineers are still still locked on flights. So, but, no, that that's the growth that's the growth path we see for us. With a largely use, US focused customer base, though, flights are still the biggest because the rail system in the US is pretty unless you're traveling on the East Coast, pretty spotty. And then so you know, and then what we haven't we we've leaned to get people into Europe thus far, but not, like, what you do once you're there yet. So And in your experience
157
00:33:09.210 --> 00:33:14.525
of coming to going, helping them rebuild their data stack,
158
00:33:14.905 --> 00:33:15.405
reimagine
159
00:33:15.785 --> 00:33:22.445
the way that their overall system architecture operates? What are some of the most interesting or unexpected or challenging lessons that you learned in the process?
160
00:33:22.905 --> 00:33:23.725
Data platform
161
00:33:24.105 --> 00:33:56.169
cutovers are always a hustle. Like, no matter how well you plan or how well you do everything, there's always this awkward phase of doing the last mile and cutting over from one system to another and then all the errors that that arise from that in terms of, like, dashboards and and business critical stuff that kinda collapses. You know, we we as an exercise with, the migration from from Snowflake to to Iceberg, we target our schemas too. We get a schema overhaul in parallel because, like, we thought some of our schemas were were inefficient for where we saw the business going. And so, like, doing all that simultaneously
162
00:33:56.549 --> 00:34:24.720
was a big project, and it always turns into a mess at the end. I I would love to figure out a better way to do that. I just I haven't yet. And so that's been interesting. And then and then there's the customer education because we did move from, preset to quick site. We were using the the
superset under the covers for a while, but decided to keep everything in the AWS stack. Because I think, you know, I think Amazon queue is interesting. I think the NLP stuff they're doing with Bedrock is interesting. My eventual goal is to get nontechnical users to be able to actually describe and define dashboards,
163
00:34:25.339 --> 00:34:29.680
and have good enough schema that it could probably handle that kind of specification,
164
00:34:30.140 --> 00:34:41.895
and that's a work in progress. It's still not still not there. I just have to kinda know what to do in the bot to a certain extent. But we already talked a little bit about some of the plans that you have for the forward looking goals of the going data architecture.
165
00:34:42.515 --> 00:34:48.375
But as somebody who is operating in the data space, what are some of the overarching
166
00:34:48.755 --> 00:34:50.295
trends or technologies
167
00:34:50.915 --> 00:34:51.415
or,
168
00:34:51.930 --> 00:35:00.750
just areas of personal interest that you're focused on and keeping an eye towards as you continue to lead and grow and scale your data operations?
169
00:35:01.130 --> 00:35:16.275
You know, having built stuff like this from scratch and then looking at where we are today with just the sheer amount of options in market for virtually every kind of data solution that is, like, wildly more scalable than what we're using, let's say, like, 6 or 8 years ago, has been really,
170
00:35:16.674 --> 00:35:30.910
really awesome to see. And so for me, like, building something this big and managing this volume of data without the tools we have today would be would be impossible for a team my size to actually manage and handle and operate. You know? And so for me, like, that's been a huge differentiator
171
00:35:31.665 --> 00:35:52.070
in market. Just the sheer amounts of, like, of concurrent data you can stream in and it not be a big deal, the way you can land and process that data at, you know, at at the multi petabyte scale and that not be a big deal. Like, those things were a big deal at other phases in my career, and so, like, that's been awesome to see from that perspective. You know, it's been kind of and, you know, even, you know,
172
00:35:52.630 --> 00:36:11.845
having to build, you know, service frameworks to serve different algorithms up and do my own custom AB. It's like, all that you can get a lot of that out of the box, and those are pretty large engineering challenges. Like, those are, like, employing a team full time to work on those sorts of things. And so that's been that's been super cool, that you can actually do a lot of stuff with not a huge amount of humans.
173
00:36:12.220 --> 00:36:17.280
And you can do ambitious stuff with not a huge amount of humans required. It's been a pretty big unlock for engineering.
174
00:36:17.580 --> 00:36:22.880
Are there any other aspects of the work that you're doing at going the data architecture
175
00:36:23.180 --> 00:36:24.720
and
platform architecture
176
00:36:25.020 --> 00:36:26.560
that you have established
177
00:36:27.085 --> 00:36:28.465
or the overall applications
178
00:36:28.845 --> 00:36:44.299
of the Trino and Iceberg combination that we didn't discuss yet that you'd like to cover before we close out the show? One of the things I like about Trino and Galaxy in general is that they are trying to build a cohesive Iceberg management solution. You know, I think with Tabula getting snapped up by Databricks,
179
00:36:44.599 --> 00:37:28.395
I think, but, you know, and it's still sort of be, like, promoting Delta under the covers. I think, like, for me, being able to work with an open lake house architecture and actually, like, work with Trino as a technology. Because Trino Trino is a great technology, but it used to be kind of a pain to manage all those clusters. I'd say, like, like, Galaxy has has provided a lot of transparency on that, and they provide some data management capabilities on top of it. I think data management is an area that kind of lacks in a lot of tool sets that I've worked within the past. Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today. Oh, man.
180
00:37:30.055 --> 00:37:35.675
Data quality is still an issue. Quality checking, it's still an unsolved problem. You know, I think,
181
00:37:36.135 --> 00:37:41.890
unless a feed completely breaks, like, it's tough to detect bugs in data feeds accurately. You know, I think
182
00:37:42.270 --> 00:37:45.250
application of AI potentially in that space is very interesting.
183
00:37:45.710 --> 00:37:54.305
And I'd say, like, data education and lineage is still really hard, especially as companies become more business derived and driven. Like, it's still tough to kind of articulate.
184
00:37:54.605 --> 00:38:03.345
You should rely on this dataset because it is concrete and transactional. Like, it it integrates with our payment processor and that is physical money in our bank account versus
185
00:38:03.690 --> 00:38:08.109
this is a tool that is hooked up to our website and is about 20% inaccurate.
186
00:38:08.650 --> 00:38:38.330
And, like, you should use it for directional data, but not hard business decisions. You know, but, like, education on that and marking that and when people like, it's just it's tough to propagate that context to customers is what I would say. And so for me, like, that's still an area that we have to invest in and solve as a business. That and, like, data leakage, like, people are still leaking PII and and, like, the governance side. So it's it's all it's it just it's it's the nontechnical probably portion of the and the policy and quality portions of data management that I still think are,
187
00:38:38.710 --> 00:38:39.210
challenges.
188
00:38:39.590 --> 00:38:47.565
Absolutely. Well, thank you very much for taking the time today to join me and share your work of building this data architecture
189
00:38:47.945 --> 00:39:01.230
and the ways that you are using Iceberg for streaming ingest. Definitely a very interesting problem space, interesting solution that you've developed. So thank you again for taking the time, and I hope you enjoy the rest of your day. Thank you. Thanks for having me. This is awesome.
190
00:39:08.490 --> 00:39:12.415
Thank you for listening, and don't forget to check out our other shows. Podcast.netcoversthepythonlanguage,
191
00:39:14.075 --> 00:39:21.775
its community, and the innovative ways it is being used, and the AI Engineering Podcast is your guide to the fast moving world of building AI systems.
192
00:39:22.395 --> 00:39:32.160
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email host at data engineering podcast.com
193
00:39:32.620 --> 00:39:38.800
with your story. Just to help other people find the show, please leave a review on Apple Podcasts and to tell your friends and coworkers.