Designing Data Platforms For Fintech Companies

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 2:30:05 PM
Duration: 2876.81
Channels: 1
1
00:00:11.264 --> 00:00:15.445
Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
2
00:00:16.384 --> 00:00:18.164
Introducing RudderStack Profiles.
3
00:00:18.785 --> 00:00:28.630
RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable enriched data to every downstream team.
4
00:00:29.255 --> 00:00:35.595
You specify the customer traits, then profiles runs the joints and computations for you to create complete customer profiles.
5
00:00:36.370 --> 00:00:39.510
Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack.
6
00:00:42.370 --> 00:01:04.730
You shouldn't have to throw away the database to build with fast changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades old batch computation model for an efficient incremental engine to get complex queries that are always up to date. With Materialise, you can. It's the only true SQL streaming database built from the ground up to meet the needs of modern data products.
7
00:01:05.525 --> 00:01:16.970
Whether it's real time dashboarding and analytics, personalization and segmentation, or automation and alerting, Materialise gives you the ability to work with fresh, correct, and scalable results, all in a familiar SQL interface.
8
00:01:17.350 --> 00:01:18.890
Go to data engineering podcast.com/materialize
9
00:01:20.470 --> 00:01:22.090
today to get 2 weeks free.
10
00:01:22.595 --> 00:01:30.534
Your host is Tobias Macy, and today I'm interviewing Andre Korchak about how to manage data in a Fintech environment. So, Andre, can you start by introducing yourself?
11
00:01:31.550 --> 00:01:37.890
Sure. Thanks for having me here. So my name is Andre. I'm a chief technical officer in the Fintech company,
12
00:01:38.545 --> 00:01:40.064
Mennite. We're working in,
13
00:01:40.625 --> 00:01:41.125
Fintech
14
00:01:41.665 --> 00:01:43.025
API space. So we,
15
00:01:43.905 --> 00:01:44.564
our company,
16
00:01:44.945 --> 00:01:45.720
is kind of
17
00:01:46.440 --> 00:01:47.580
it's like AWS,
18
00:01:48.360 --> 00:01:50.940
services for those who want to create Fintech applications,
19
00:01:51.320 --> 00:01:53.340
but instead of supplying our customers,
20
00:01:54.215 --> 00:01:56.715
with, low level components like databases
21
00:01:57.015 --> 00:02:10.635
or cloud service, we basically provide them the building blocks like invoicing or accounts or payments engine. So instead of writing all the business logic, you can plug in, all our p all APIs in, and they're good to go.
22
00:02:11.194 --> 00:02:13.135
That's that's me, and then that's our product.
23
00:02:13.915 --> 00:02:20.095
So, obviously, I'm a technical director, and, I'm responsible for everything, we're doing in our company.
24
00:02:20.739 --> 00:02:23.959
So taking care about data is part of my responsibilities.
25
00:02:24.739 --> 00:02:28.200
I'm not actively writing data processing pipelines anymore
26
00:02:28.825 --> 00:02:30.685
or I'm not, managing
27
00:02:31.705 --> 00:02:39.250
data storages and data warehouses directly, but I know quite well how everything works, and, I'm I'm responsible for all decisions we take.
28
00:02:39.709 --> 00:02:42.290
And do you remember how you first got started working in data?
29
00:02:44.075 --> 00:02:44.575
Sure.
30
00:02:45.275 --> 00:02:48.415
I have solid backgrounds in software engineering engineering
31
00:02:48.715 --> 00:02:51.295
almost 24 years since I started
32
00:02:52.314 --> 00:02:53.740
writing code. And,
33
00:02:54.680 --> 00:02:58.380
my previous company, my previous startup was, an EdTech company,
34
00:02:59.000 --> 00:03:01.500
and, we were doing natural language processing.
35
00:03:02.315 --> 00:03:02.815
Obviously,
36
00:03:03.435 --> 00:03:06.415
machine learning was involved, and, I was doing
37
00:03:07.995 --> 00:03:17.800
data management, data processing, all these kind of things. Now I'm working in Fintech space, and, we're not machine learning savvy, but, we still have to process customer data.
38
00:03:18.185 --> 00:03:23.405
They still have to train some algorithms, and, obviously, I'm involved in into all these things.
39
00:03:24.185 --> 00:03:27.405
And now getting into the Fintech
40
00:03:27.750 --> 00:03:29.849
specifics and some of the challenges
41
00:03:30.709 --> 00:03:35.769
around data in that space, I'm wondering if you can just start by giving a bit of a summary of
42
00:03:36.135 --> 00:03:42.795
the ways that data is used in Fintech and some of the particular challenges in that space pertaining to data.
43
00:03:44.440 --> 00:03:54.765
Sure. So our company is working with small and medium businesses, and, the space is a bit different from, say, private banking or investment banking. So my examples will be relevant,
44
00:03:55.385 --> 00:03:56.125
only for,
45
00:03:56.825 --> 00:04:00.890
SMEs, and, we have few challenges. So challenge number 1,
46
00:04:01.430 --> 00:04:03.610
we have to analyze and and interpret
47
00:04:04.150 --> 00:04:06.970
quite a lot of financial documents. I'm talking about invoicing,
48
00:04:07.670 --> 00:04:08.170
invoices,
49
00:04:08.645 --> 00:04:11.545
cancellation notes, purchase orders, receipts,
50
00:04:11.845 --> 00:04:12.345
contracts.
51
00:04:13.045 --> 00:04:13.705
So, basically,
52
00:04:14.565 --> 00:04:16.104
in our case, we have to
53
00:04:16.750 --> 00:04:18.610
do OCR. We have to do,
54
00:04:19.230 --> 00:04:20.930
pattern recognition and,
55
00:04:21.389 --> 00:04:23.730
other things related to document processing.
56
00:04:24.085 --> 00:04:26.745
Now second thing, we have to do document categorization.
57
00:04:27.045 --> 00:04:29.065
So we have to be able to distinguish
58
00:04:29.445 --> 00:04:34.636
invoices from cancellation notes and purchase orders and surprise. Those documents look
59
00:04:35.500 --> 00:04:37.259
very similar to each other, and,
60
00:04:37.979 --> 00:04:39.599
separating this this
61
00:04:40.354 --> 00:04:43.175
files in this document. It's indeed very hard task.
62
00:04:43.555 --> 00:04:49.255
And third thing, we have to deal with compliance. And, in order to do this, we're doing fraud prevention.
63
00:04:49.960 --> 00:04:51.340
We do transaction monitoring,
64
00:04:51.640 --> 00:04:54.860
and, other things that might be relevant to that topic.
65
00:04:55.400 --> 00:04:59.020
So these are main things we do in, in our company,
66
00:04:59.845 --> 00:05:01.945
and data related things we do in our company.
67
00:05:03.365 --> 00:05:04.185
Now, well,
68
00:05:04.485 --> 00:05:06.745
a few words about the data sources,
69
00:05:07.445 --> 00:05:11.030
we we have. Obviously, we have, financial operations.
70
00:05:11.729 --> 00:05:14.790
Talking about, bank transactions or card payments.
71
00:05:15.570 --> 00:05:15.925
This
72
00:05:16.405 --> 00:05:18.185
this type of data is extremely,
73
00:05:18.645 --> 00:05:21.384
well, strike well, not extremely, but it's well structured.
74
00:05:22.085 --> 00:05:28.280
Yes. There are tiny differences like a different data formats or different currency codes. But, when you have,
75
00:05:28.840 --> 00:05:31.660
a list of, transactions made from the bank account,
76
00:05:32.200 --> 00:05:33.820
you can make pretty good guesses
77
00:05:34.895 --> 00:05:36.115
where where to find
78
00:05:37.055 --> 00:05:39.235
transaction descriptions or card numbers,
79
00:05:39.775 --> 00:05:41.875
and, that's not very challenging.
80
00:05:42.330 --> 00:05:42.990
Well, documents.
81
00:05:43.370 --> 00:05:45.630
Financial documents coming from customers,
82
00:05:46.010 --> 00:05:48.590
and, there are all kinds of documents,
83
00:05:49.530 --> 00:05:50.590
PDF files,
84
00:05:51.335 --> 00:05:51.835
photos,
85
00:05:52.295 --> 00:05:52.795
scans,
86
00:05:53.735 --> 00:05:54.795
or bank statements,
87
00:05:55.335 --> 00:05:55.835
receipts,
88
00:05:56.615 --> 00:05:57.115
and
89
00:05:57.495 --> 00:06:00.330
dealing with these docs is is is indeed very hard.
90
00:06:00.889 --> 00:06:04.110
Now, we have some data that is not directly
91
00:06:04.569 --> 00:06:05.629
relevant to,
92
00:06:06.009 --> 00:06:06.509
financial,
93
00:06:07.370 --> 00:06:10.575
transactions, but this data is is very helpful,
94
00:06:11.515 --> 00:06:13.295
when you're doing finance management.
95
00:06:13.835 --> 00:06:14.655
We're talking about
96
00:06:15.595 --> 00:06:17.410
CRM records or,
97
00:06:18.110 --> 00:06:22.370
information from the marketing campaign about marketing budgets or
98
00:06:22.830 --> 00:06:26.210
customer support requests regarding some financial operations.
99
00:06:28.895 --> 00:06:33.715
And last source of information we did dealing with, is internal data. We're talking about,
100
00:06:34.175 --> 00:06:35.315
logs or
101
00:06:35.979 --> 00:06:37.599
conversations with our customers,
102
00:06:38.460 --> 00:06:40.960
streams of events generated by the users,
103
00:06:41.580 --> 00:06:42.960
and other internal things.
104
00:06:43.824 --> 00:06:48.225
This this this data is not feasible to to our customers, but,
105
00:06:48.945 --> 00:06:54.310
it it plays critical role, when we operate in our software and then making business decisions.
106
00:06:55.009 --> 00:06:59.910
In terms of the application of data in the business context for fintechs,
107
00:07:00.289 --> 00:07:02.470
I'm wondering what are some of the
108
00:07:02.805 --> 00:07:03.705
key capabilities
109
00:07:04.085 --> 00:07:07.625
that are powered by data in a fintech context
110
00:07:08.165 --> 00:07:13.199
and some of the complexities that are involved in being able to bring data to bear for those problems?
111
00:07:14.380 --> 00:07:15.340
Sure. So,
112
00:07:15.820 --> 00:07:16.320
transactions,
113
00:07:17.100 --> 00:07:20.160
and, all the financial operations. As I mentioned before,
114
00:07:20.975 --> 00:07:22.575
this data is is,
115
00:07:22.975 --> 00:07:27.395
well structured, so it it's kind of easy to deal with financial records.
116
00:07:28.310 --> 00:07:29.770
There is no need in sophisticated
117
00:07:30.070 --> 00:07:34.565
algorithms or machine learning because we already have all the numbers in place, so we can just
118
00:07:35.045 --> 00:07:36.085
do do the math,
119
00:07:36.645 --> 00:07:41.065
or these records. So it's it's not complicated, but but it's it's crucial to,
120
00:07:41.445 --> 00:07:46.300
provide to our customers the spacing functionality related to, financial transactions.
121
00:07:47.000 --> 00:07:49.020
Settlement management. It's something
122
00:07:49.320 --> 00:07:49.820
that's
123
00:07:51.695 --> 00:07:56.835
it's kind of hard to deal with this because as I mentioned before, there are tons of different documents. And,
124
00:07:57.535 --> 00:07:59.930
in the modern world world, people still,
125
00:08:01.290 --> 00:08:03.070
use, invoices or contracts
126
00:08:03.610 --> 00:08:04.110
printed
127
00:08:04.650 --> 00:08:05.310
on paper,
128
00:08:05.930 --> 00:08:11.985
and it's it's extremely hard to deal with the these records before before you'll be able to to do something with these,
129
00:08:12.385 --> 00:08:20.240
documents. You have to scan them, you have to verify them, and you have to categorize these documents. This is called preparatory accounting,
130
00:08:20.699 --> 00:08:23.120
and it's it's extremely challenging area.
131
00:08:23.595 --> 00:08:27.935
Obviously, it it's pretty exhausting to deal with all these documents because,
132
00:08:30.075 --> 00:08:30.575
sometimes
133
00:08:30.955 --> 00:08:33.240
they they come in from different channels,
134
00:08:33.620 --> 00:08:36.839
in different from different countries. They may look different,
135
00:08:37.300 --> 00:08:41.319
and, it's it's for me knowing to categorize these documents to assign
136
00:08:41.935 --> 00:08:45.315
understand these documents to correct department, to right department. So,
137
00:08:45.775 --> 00:08:48.015
that's what we're doing. We're processing all,
138
00:08:48.790 --> 00:08:52.970
all the flows, informations coming on paper, via fax, via email,
139
00:08:53.670 --> 00:08:54.390
and we,
140
00:08:54.870 --> 00:08:57.370
we categorize, recognize these documents, and
141
00:08:57.774 --> 00:08:59.154
head into the right direction.
142
00:08:59.615 --> 00:09:02.255
And analytics, it's something that comes on top of,
143
00:09:02.815 --> 00:09:05.235
transaction information and on top of documents.
144
00:09:05.610 --> 00:09:07.230
So we have to be able to calculate,
145
00:09:07.690 --> 00:09:10.590
the cash flow. We have to be able to identify
146
00:09:11.210 --> 00:09:13.790
gaps in the cash flow. We have to do some forecasting,
147
00:09:14.795 --> 00:09:21.675
and other things. So so it's basically 3, 3 major areas we're operating in, documents management, transaction,
148
00:09:22.155 --> 00:09:24.255
transactions management, and analytics.
149
00:09:25.339 --> 00:09:27.519
I know that in the Fintech sector,
150
00:09:27.820 --> 00:09:29.519
there are a lot of regulatory
151
00:09:29.899 --> 00:09:30.399
requirements.
152
00:09:30.700 --> 00:09:36.465
The risks of getting things wrong in the business are quite high because you can start to lose a lot of money very quickly.
153
00:09:37.005 --> 00:09:37.825
And, also,
154
00:09:38.205 --> 00:09:44.340
any errors in the data or in the application of data can lead to a loss of trust in,
155
00:09:44.720 --> 00:09:52.325
among the customers, which can also lead to a pretty substantial financial hit. And I'm wondering what are some of the ways that those regulatory
156
00:09:52.785 --> 00:09:53.925
and trust issues
157
00:09:54.305 --> 00:09:55.685
factor into the
158
00:09:56.065 --> 00:10:00.680
ways that organizations think about the collection of an application of data?
159
00:10:01.220 --> 00:10:04.360
So as a as a fintech company, we have to
160
00:10:04.975 --> 00:10:06.515
obtain, ISO 2701
161
00:10:07.135 --> 00:10:08.755
certificate. The certificates
162
00:10:10.095 --> 00:10:11.875
defining, the way we are processing
163
00:10:12.760 --> 00:10:15.339
critical financial information. Well, not only financial.
164
00:10:15.720 --> 00:10:20.220
Everything that's, that is crucial for our customers, including scans of IDs,
165
00:10:21.055 --> 00:10:22.515
receipts, and transactions,
166
00:10:23.295 --> 00:10:24.195
all these things.
167
00:10:25.295 --> 00:10:27.235
In in order to deal with all these things,
168
00:10:27.855 --> 00:10:29.315
we have to be ISO certified.
169
00:10:30.250 --> 00:10:39.149
By by the law, when you obtain ISO certificate, you have to enforce some policies on engineers, on the dev ops team, and on all the managers.
170
00:10:39.584 --> 00:10:42.245
So the, the access to the data is restrict,
171
00:10:43.024 --> 00:10:43.845
and only,
172
00:10:44.464 --> 00:10:49.150
a few approved and certified engineers inside of the company can have access to this.
173
00:10:49.710 --> 00:10:50.110
And,
174
00:10:50.670 --> 00:10:51.970
couple of DevOps engineers,
175
00:10:52.430 --> 00:11:01.085
obviously, analysts and managers also can have the access to to the data, but all the critical data fields, they they masked and removed or an anonymized.
176
00:11:01.865 --> 00:11:02.365
And
177
00:11:03.545 --> 00:11:14.810
we have some, data protection policies, backup policies, disaster recovery policies. So, basically, we have policies for everywhere. And we also have to make sure that we not only have these policies, but we're able to,
178
00:11:15.695 --> 00:11:16.195
identify,
179
00:11:16.815 --> 00:11:20.514
let's say, data leakages or critical disasters happening in infrastructure.
180
00:11:21.375 --> 00:11:21.875
And
181
00:11:22.790 --> 00:11:23.930
we do practice,
182
00:11:24.310 --> 00:11:27.290
we do run some training exercises with engineers
183
00:11:27.830 --> 00:11:30.010
and trying to simulate disasters
184
00:11:30.390 --> 00:11:30.890
or
185
00:11:31.905 --> 00:11:35.285
infrastructure cluster collapse and see what what happens after.
186
00:11:35.825 --> 00:11:38.325
Obviously, you have backups for backups, and
187
00:11:39.400 --> 00:11:41.580
it's not because we want this, but because,
188
00:11:42.440 --> 00:11:43.500
we must have,
189
00:11:44.600 --> 00:11:48.620
backups even for backups. And, all the all the backups are encrypted
190
00:11:49.135 --> 00:11:51.635
and, not accessible by third party
191
00:11:52.015 --> 00:11:57.235
companies, even AWS administrators won't be won't be able to get access to to our data.
192
00:11:58.139 --> 00:11:58.800
So, basically,
193
00:11:59.180 --> 00:12:00.560
all the all the cybersecurity,
194
00:12:02.300 --> 00:12:05.519
all the all best cybersecurity practice are in place.
195
00:12:05.884 --> 00:12:12.144
Also, we have multiple data centers, and they have we have separate cloud infrastructures there. So
196
00:12:12.685 --> 00:12:19.190
people from from 1 when 1 data center cannot can cannot get access to the data restoring another data center.
197
00:12:20.050 --> 00:12:25.315
That's partially because we have to be compliant with GDPR and, other regional regulations, but,
198
00:12:25.935 --> 00:12:29.075
also, it's yet another security measure, and
199
00:12:29.930 --> 00:12:35.790
we have to make hackers' life quite miserable who they will somehow get access to to what we have.
200
00:12:36.650 --> 00:12:37.150
Also,
201
00:12:37.885 --> 00:12:45.745
we have to be very cautious when we're choosing third party providers. As a fintech company, we have to do background checks, and we have to
202
00:12:46.620 --> 00:12:47.920
go to the documentation,
203
00:12:48.220 --> 00:12:49.360
and we have to be,
204
00:12:49.740 --> 00:12:55.665
very cautious when we choose, our service providers. So they have they must be ISO 2701
205
00:12:56.204 --> 00:13:01.185
certified because, otherwise, we're, we won't be able to maintain our certification.
206
00:13:02.170 --> 00:13:03.070
They have to
207
00:13:04.170 --> 00:13:06.750
be regulated by local laws and regulations.
208
00:13:07.370 --> 00:13:11.505
And, also, when we're processing, the data with third party providers, we have to,
209
00:13:12.145 --> 00:13:12.625
include,
210
00:13:13.025 --> 00:13:22.839
some closures into, terms and conditions because without written consent, we cannot do absolutely anything, and we have to inform our users that, their data will be used
211
00:13:23.220 --> 00:13:25.720
by a third party machine learning
212
00:13:26.100 --> 00:13:27.639
company that will be doing,
213
00:13:28.019 --> 00:13:29.079
invoice processing.
214
00:13:30.645 --> 00:13:31.145
Basically,
215
00:13:31.925 --> 00:13:32.904
working in Fintech
216
00:13:33.925 --> 00:13:37.785
forces you to sign and to maintain lots of contracts,
217
00:13:38.760 --> 00:13:42.700
agreements even with your own employees, with DevOps team, with engineers.
218
00:13:43.720 --> 00:13:49.334
They cannot have access literally to anything unless sign the contract, and that's pretty annoying.
219
00:13:50.115 --> 00:13:50.615
Basically,
220
00:13:51.634 --> 00:13:52.935
dealing with those things,
221
00:13:53.555 --> 00:14:02.475
well, definitely makes sense, but it slows business down because before we can take any actions, we have to make sure that we're good on the legal side. And before we can,
222
00:14:03.675 --> 00:14:08.975
start working on new feature or before we can sign a contract with new counterpart or vendor,
223
00:14:09.834 --> 00:14:12.175
our legal team have to review it, and
224
00:14:13.020 --> 00:14:14.720
it's basically just pre annoying.
225
00:14:15.100 --> 00:14:25.005
Apart of that, we are we are not very different from other software companies. We just extremely have it regulated, and we have backups to backups because we have to make sure that,
226
00:14:25.644 --> 00:14:27.505
we won't lose any customer data.
227
00:14:27.884 --> 00:14:33.300
To that point of all of your service providers requiring ISO 2701
228
00:14:34.320 --> 00:14:34.820
certification
229
00:14:35.440 --> 00:14:37.460
and the strict
230
00:14:37.795 --> 00:14:41.255
security requirements around the data and the ways that it's processed,
231
00:14:41.714 --> 00:14:42.774
how does that influence
232
00:14:43.075 --> 00:14:53.620
the typical build versus buy decision when you're figuring out what your architectural and system components are going to be? What are the different data flows that you're going to support?
233
00:14:54.725 --> 00:14:55.225
Okay.
234
00:14:55.765 --> 00:14:57.065
That's interesting question,
235
00:14:57.524 --> 00:14:59.545
because when you're working in Fintech,
236
00:15:00.005 --> 00:15:10.800
you cannot build MVP. Well, yes, you definitely can, build a software that does something, for your customers, but at the same time, you have to be compliant, and you have to follow all these
237
00:15:11.475 --> 00:15:12.855
laws, rules, and regulations.
238
00:15:13.475 --> 00:15:18.615
Means that you have to invest quite a lot of time into hiring and training people, into
239
00:15:19.790 --> 00:15:23.569
signing all these contracts, and establishing all these data protection policies.
240
00:15:24.110 --> 00:15:26.370
It means that in our in order
241
00:15:26.750 --> 00:15:31.075
to to get to Fintech space, you have to pay quite a lot for entry ticket.
242
00:15:31.615 --> 00:15:33.555
Now if you're a Fintech company,
243
00:15:34.895 --> 00:15:38.195
you probably will try to buy as many components,
244
00:15:38.710 --> 00:15:41.210
APIs, and services as possible because,
245
00:15:42.150 --> 00:15:47.690
they're usually cheaper than, doing, all these things in house. If you if you're
246
00:15:48.555 --> 00:15:52.975
a seed stage company, you cannot afford to have a compliance officer
247
00:15:53.435 --> 00:15:53.935
or,
248
00:15:54.955 --> 00:15:56.175
for DevOps engineer,
249
00:15:56.714 --> 00:15:57.615
available 247.
250
00:15:59.339 --> 00:16:00.880
But you can pay 10,
251
00:16:01.339 --> 00:16:03.500
15, 20, 000 for that, and,
252
00:16:04.060 --> 00:16:05.839
that sounds like a reasonable price
253
00:16:06.365 --> 00:16:14.850
comparing to alternatives, and alternatives is to hire everyone, to train their people, to find managers, to write product specs, to establish all these,
254
00:16:15.410 --> 00:16:23.110
data regulation policies. So, basically, even even simple invoicing solution, if you want to build this in house, might cost you a few million. And,
255
00:16:25.035 --> 00:16:26.575
MVP of invoicing
256
00:16:26.955 --> 00:16:29.855
engine doesn't seem like a extremely sophisticated,
257
00:16:30.795 --> 00:16:31.295
software,
258
00:16:32.360 --> 00:16:34.600
but it could be because of these,
259
00:16:35.080 --> 00:16:36.380
laws, rules, and regulations.
260
00:16:37.000 --> 00:16:42.755
So if you're early stage company and you want to enter Fintech space, you definitely have to consider,
261
00:16:44.175 --> 00:16:45.875
service providers that's already,
262
00:16:46.495 --> 00:16:47.955
work in this space because
263
00:16:48.415 --> 00:16:53.949
you don't have any other alternatives. Now if you're a big company, at this point of time, you already can
264
00:16:56.009 --> 00:17:01.214
afford to build something in house, and there are usually a couple main reasons for that. So
265
00:17:01.595 --> 00:17:07.260
maybe you already have experienced team and you have all the expertise that is required in order to build
266
00:17:08.060 --> 00:17:13.500
things you want to build, in this case, yes, you can afford it. Another thing, if you already have,
267
00:17:14.700 --> 00:17:16.460
established business and you have,
268
00:17:17.100 --> 00:17:21.665
and you're already pumping in significant volumes through your your software. In this case,
269
00:17:22.205 --> 00:17:22.705
probably,
270
00:17:23.325 --> 00:17:25.985
building your own solution will be cheaper
271
00:17:26.285 --> 00:17:26.785
than
272
00:17:27.380 --> 00:17:30.200
buying third party solution simply because of their wallets.
273
00:17:30.900 --> 00:17:31.640
Or maybe,
274
00:17:32.820 --> 00:17:34.760
signing contract with third party
275
00:17:36.165 --> 00:17:39.625
service provider is contradicting to your business strategy. And,
276
00:17:40.565 --> 00:17:45.350
yes, building a new fintech vertical inside of your company will be expensive,
277
00:17:45.650 --> 00:17:46.790
but you have to,
278
00:17:47.810 --> 00:17:50.790
but you have to make this, choice in order to
279
00:17:51.170 --> 00:17:53.270
make your long term strategy successful.
280
00:17:54.095 --> 00:17:57.955
If you're not a big business, you you don't have any strategic goals,
281
00:17:58.495 --> 00:18:00.595
my suggestion will be to consider,
282
00:18:01.215 --> 00:18:04.020
for third party service providers for your Fintech
283
00:18:04.640 --> 00:18:05.140
vertical.
284
00:18:06.080 --> 00:18:09.060
But, yes, it creates vendor lock, but the alternative,
285
00:18:09.600 --> 00:18:10.915
it can be left
286
00:18:11.795 --> 00:18:17.335
away out of the Fintech market because, as I've said before, the entry ticket is extremely expensive.
287
00:18:18.250 --> 00:18:21.550
And going back as well to your point of requiring
288
00:18:21.930 --> 00:18:23.870
very rigorous backup capabilities,
289
00:18:24.170 --> 00:18:24.910
I'm wondering
290
00:18:26.010 --> 00:18:26.990
what are the
291
00:18:27.365 --> 00:18:35.385
orders of magnitude in terms of the size of the data that you're dealing with and some of the ways that you have to think about managing backups
292
00:18:35.789 --> 00:18:37.890
and in particular being able to
293
00:18:38.270 --> 00:18:45.695
maybe use backup strategies that allow for just using compressed deltas as opposed to having to do multiple full copies the data?
294
00:18:46.555 --> 00:18:50.235
So transactional data doesn't take lots of space because,
295
00:18:50.635 --> 00:18:52.255
we're just talking about numbers
296
00:18:52.715 --> 00:18:54.630
and very short strings of text.
297
00:18:55.510 --> 00:18:58.090
There is no problem with transactional data. And,
298
00:18:58.710 --> 00:18:59.210
usually,
299
00:19:00.870 --> 00:19:03.610
Fintech companies not processing payments directly.
300
00:19:04.365 --> 00:19:13.985
Usually, there is a second copy of information that is always available on on the side of the payment provider. That's easy part. Now we're talking about the fun part, financial documents.
301
00:19:16.270 --> 00:19:19.490
Well, financial documents, they have significant sizes.
302
00:19:20.350 --> 00:19:26.804
Something well, typical invoice, you know, could be a could have a size of 1 megabyte or an average.
303
00:19:27.265 --> 00:19:35.110
And, some companies, they're having, like, a multipage long contracts, and and the size of those documents can be up to,
304
00:19:35.810 --> 00:19:40.055
up to, like, a half gigabyte, and, obviously, we have to take care about that.
305
00:19:40.595 --> 00:19:50.520
And that's a place when we have to rely heavily on on the cloud infrastructure, and the, s 3 storage is here to save the day, obviously, but we cannot give direct access,
306
00:19:51.059 --> 00:19:56.040
to, to the buckets stored inside of s 3. So, usually, we have to
307
00:19:56.835 --> 00:20:01.575
store and create its binary data inside of s 3, and we have to stream data
308
00:20:01.955 --> 00:20:06.135
from those buckets to to our customers. Now talking about backups,
309
00:20:06.660 --> 00:20:10.520
basically, we created not very sophisticated, but the same time reliable,
310
00:20:11.860 --> 00:20:12.580
system of,
311
00:20:13.060 --> 00:20:13.880
that source,
312
00:20:14.420 --> 00:20:16.475
of files in s 3, Basically,
313
00:20:17.575 --> 00:20:19.595
using cross regional replications
314
00:20:19.895 --> 00:20:21.434
of AWS, obviously.
315
00:20:21.815 --> 00:20:23.995
Also, we're making secure backups
316
00:20:25.340 --> 00:20:26.080
via other
317
00:20:26.460 --> 00:20:37.635
AWS services like AWS Glass here, which which which is a perfect solution for this kind of things. But we have to make sure that those binary files, they're encrypted, and they're not, available,
318
00:20:37.935 --> 00:20:43.555
outside of our private network. If our customers want to get access to binary data, we have to
319
00:20:44.660 --> 00:20:48.440
stream and decrypt, these files on the fly directly to our customers.
320
00:20:51.300 --> 00:20:52.440
Data lakes are notoriously
321
00:20:52.820 --> 00:20:53.320
complex.
322
00:20:53.845 --> 00:21:09.490
For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte scale SQL analytics fast at a fraction of the cost of traditional methods so that you can meet all of your data needs ranging from AI to data applications to complete analytics.
323
00:21:10.005 --> 00:21:13.465
Trusted by teams of all sizes, including Comcast and DoorDash,
324
00:21:13.765 --> 00:21:20.160
Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises.
325
00:21:20.860 --> 00:21:27.520
And Starburst does all of this on an open architecture with first class support for Apache Iceberg, Delta Lake, and Hoody,
326
00:21:27.855 --> 00:21:30.034
so you always maintain ownership of your
327
00:21:30.735 --> 00:21:33.715
data. Want to see Starburst in action? Go to dataengineeringpodcast.com/starburst
328
00:21:36.060 --> 00:21:42.640
and get $500 in credits to try Starburst Galaxy today, the easiest and fastest way to get started using Trino.
329
00:21:44.534 --> 00:21:46.955
In terms of the application of data
330
00:21:47.495 --> 00:21:49.274
in the Fintech sector,
331
00:21:49.654 --> 00:21:56.500
they have been 1 of the longest users of machine learning in the context of things like fraud detection,
332
00:21:57.360 --> 00:22:06.515
particularly in banking contexts. And I'm wondering what are some of the primary ways that you're seeing ML being applied in the Fintech sector
333
00:22:07.055 --> 00:22:08.915
and some of the ways that those,
334
00:22:09.570 --> 00:22:10.710
ML requirements
335
00:22:11.010 --> 00:22:18.390
as far as training and serving influence the architectural design and the capabilities of data platforms for those types of applications?
336
00:22:19.465 --> 00:22:21.725
Well, obviously, after OpenAI,
337
00:22:22.905 --> 00:22:28.045
released the chat GPT, everyone starts started worried about, their future in the business.
338
00:22:28.700 --> 00:22:29.100
And,
339
00:22:29.580 --> 00:22:33.600
I had some conversations with our product managers and other c level managers,
340
00:22:34.220 --> 00:22:37.515
and I wasn't really worried at all because, like, from my perspective,
341
00:22:37.815 --> 00:22:40.635
this raise of, large linguistic models and
342
00:22:42.535 --> 00:22:48.340
other AI technologies won't change Fintech industry at all. So I have a few reasons, for that.
343
00:22:48.800 --> 00:22:49.700
First of all,
344
00:22:50.240 --> 00:22:53.940
machine learning is already in Fintech for at least couple decades.
345
00:22:54.400 --> 00:22:58.645
And fraud prevention, that's that's 1 story, but, you also,
346
00:22:59.185 --> 00:23:05.045
machine learning algorithms are widely used for credit scoring or for risk analysis.
347
00:23:05.770 --> 00:23:07.230
These algorithms are
348
00:23:07.770 --> 00:23:09.310
quite old, and
349
00:23:09.930 --> 00:23:11.470
they were here even before,
350
00:23:12.090 --> 00:23:13.390
deep learning and large
351
00:23:13.905 --> 00:23:17.845
neural nets. So banks using this, their subgrains already for decades.
352
00:23:18.465 --> 00:23:28.380
And, plus, if if you're talking about transactional data, as I mentioned before, this data is extremely well structured. So we don't we don't have to run all this sophisticated algorithms,
353
00:23:28.920 --> 00:23:31.735
on financial data because we simply can do math.
354
00:23:32.615 --> 00:23:33.515
So modern,
355
00:23:34.455 --> 00:23:35.255
machine learning,
356
00:23:36.375 --> 00:23:38.554
won't change that much because
357
00:23:39.559 --> 00:23:42.700
pretty much everything was was done in this space before.
358
00:23:43.480 --> 00:23:43.980
Now,
359
00:23:44.759 --> 00:23:45.659
the big thing,
360
00:23:46.440 --> 00:23:53.195
was your optical charity recognition. As I said before, we have to deal, with documents that are coming on paper,
361
00:23:54.215 --> 00:23:58.260
and we have to deal with status of receipts, and it's green line.
362
00:23:58.640 --> 00:24:02.900
And, deep learning and other machine learning algorithms, they they definitely
363
00:24:03.760 --> 00:24:05.460
revolutionize that space because,
364
00:24:06.215 --> 00:24:09.195
the quality of OCR processing increased dramatically.
365
00:24:09.495 --> 00:24:09.995
And,
366
00:24:10.535 --> 00:24:14.870
if you remember, like, 10 years ago, there was a thing called Tesseract,
367
00:24:15.330 --> 00:24:16.370
and the quality of,
368
00:24:17.090 --> 00:24:17.330
character
369
00:24:18.530 --> 00:24:26.945
optical characters recognition was pretty poor. The accuracy was pretty low. So you could expect, like, back in the day, something around 40, 50%.
370
00:24:27.565 --> 00:24:29.804
Now with with a w s text track,
371
00:24:30.284 --> 00:24:34.610
we can have accuracy up to 95, 97 percent on some documents.
372
00:24:35.070 --> 00:24:36.909
So pretty much all the text,
373
00:24:37.710 --> 00:24:39.784
is getting recognized properly.
374
00:24:40.245 --> 00:24:42.024
And, also, the quality of preprocessing
375
00:24:42.565 --> 00:24:43.544
increased dramatically.
376
00:24:44.164 --> 00:24:46.184
So if you if you have a folder
377
00:24:46.660 --> 00:24:50.520
taken some if you have a folder of receipt that was taken
378
00:24:50.980 --> 00:24:51.800
in the dark,
379
00:24:52.260 --> 00:24:53.080
with flashlights
380
00:24:53.620 --> 00:24:54.100
and,
381
00:24:54.580 --> 00:24:59.875
with the wrong angle, the text rack will still be able to recognize and extract data
382
00:25:00.335 --> 00:25:00.815
from the,
383
00:25:01.695 --> 00:25:03.215
from that folder. So,
384
00:25:03.775 --> 00:25:04.275
OCR
385
00:25:04.735 --> 00:25:05.235
definitely,
386
00:25:06.020 --> 00:25:06.520
reshaped,
387
00:25:06.980 --> 00:25:18.775
the preparatory accounting space and the way SMEs working with financial data and financial documents right now. Another thing that is definitely affected by a rise of more in machine learning technology,
388
00:25:19.395 --> 00:25:20.615
I'm talking about machine,
389
00:25:20.995 --> 00:25:22.615
large linguistic models,
390
00:25:23.899 --> 00:25:27.840
If you're a Fintech company, normally, you have 1 support,
391
00:25:28.779 --> 00:25:29.840
customer support
392
00:25:30.695 --> 00:25:32.635
officer, 1 customer support engineer
393
00:25:33.095 --> 00:25:35.515
per 1, 000 customers. And if,
394
00:25:36.695 --> 00:25:37.435
all these
395
00:25:37.809 --> 00:25:41.649
customers will decide to write to that person at once, you basically,
396
00:25:42.130 --> 00:25:43.110
are going to have
397
00:25:43.970 --> 00:25:45.110
problems because
398
00:25:45.505 --> 00:25:47.525
this, usually, the customer support,
399
00:25:48.305 --> 00:25:53.070
team is not designed to handle big volumes. And that's what typically happens when you have
400
00:25:53.470 --> 00:26:03.315
major incidents in in banking infrastructure. People start texts immediately start texting and calling you, and your customer support center is not able to handle this request.
401
00:26:04.174 --> 00:26:05.715
So large linguistic models,
402
00:26:06.174 --> 00:26:09.715
decrease the operational cost for, customer support,
403
00:26:10.429 --> 00:26:10.929
And,
404
00:26:11.950 --> 00:26:18.289
help desk team is 1 of the biggest drivers drivers of the expenses if if you run a fintech company.
405
00:26:18.705 --> 00:26:19.205
So
406
00:26:19.745 --> 00:26:23.284
LLMs are potentially able to completely eliminate,
407
00:26:25.919 --> 00:26:28.659
well, not potentially completely eliminate, but significantly
408
00:26:28.960 --> 00:26:39.835
reduce the side of the support team. It'll be possible to have, let's say, 1, 1 agent serving 10, 000 people and 20, 000 people. And it's something that definitely
409
00:26:42.080 --> 00:26:45.539
will, will improve over the time, and we have big plans for that.
410
00:26:46.720 --> 00:26:47.220
Another
411
00:26:47.919 --> 00:26:55.085
major challenge in all data applications and data platforms, but in particular with Fintech because of the regulatory
412
00:26:55.385 --> 00:26:57.404
burdens, is that of data governance.
413
00:26:57.720 --> 00:26:58.700
And I'm wondering,
414
00:26:59.080 --> 00:27:04.780
what are some of the ways that you're approaching that problem of data governance and ensuring that you have appropriate
415
00:27:05.434 --> 00:27:08.815
visibility and access control and segmentation
416
00:27:09.195 --> 00:27:12.015
of data access and some of the
417
00:27:12.370 --> 00:27:20.015
organizational and policy aspects that you've had to invest in to be able to ensure that you're doing data governance in a way that is,
418
00:27:20.415 --> 00:27:23.155
compliant and fulfills the regulatory needs?
419
00:27:24.575 --> 00:27:32.529
Yeah. As I mentioned before, data access is extremely limited, and only, a few trained people can get can get access to this.
420
00:27:32.830 --> 00:27:34.769
Also, every team in every vertical
421
00:27:35.070 --> 00:27:38.210
in our company, they have separate database clusters.
422
00:27:38.804 --> 00:27:41.225
So they can see only only,
423
00:27:41.605 --> 00:27:44.405
the data of their users, and,
424
00:27:44.725 --> 00:27:52.669
only few people in the company can have access to all databases in our company, including me and maybe engineering, managing, couple DevOps engineers.
425
00:27:53.610 --> 00:27:55.275
Now all this we have to do,
426
00:27:55.835 --> 00:27:59.695
we have to deal with data analysis, and we have to give access to,
427
00:28:00.075 --> 00:28:02.255
to the data to analysts and managers.
428
00:28:02.850 --> 00:28:04.470
So we have to offload
429
00:28:04.850 --> 00:28:08.230
all the all the information from database to data warehouse.
430
00:28:08.850 --> 00:28:09.885
And before,
431
00:28:10.325 --> 00:28:18.025
if before we send data to the data warehouse, we have to remove or mask or anonymize critical parts of information.
432
00:28:18.649 --> 00:28:19.230
So, basically,
433
00:28:19.929 --> 00:28:21.789
analysts and managers, they can do
434
00:28:22.090 --> 00:28:24.110
they still can analyze data without
435
00:28:24.490 --> 00:28:25.470
the risk to
436
00:28:25.850 --> 00:28:27.710
accidentally leak some confidential
437
00:28:28.295 --> 00:28:30.155
information on the site. Also,
438
00:28:30.695 --> 00:28:35.595
all the configuration files on all the operations, all database clusters, they're recorded
439
00:28:35.895 --> 00:28:36.635
in repositories.
440
00:28:37.340 --> 00:28:40.160
So we're a GitOps company. And in order to do something,
441
00:28:41.660 --> 00:28:43.840
with, with production or even staging data,
442
00:28:44.365 --> 00:28:45.245
You have to,
443
00:28:45.725 --> 00:28:55.179
write your database queries in repositories. Some somebody is going to review them. And if if the code is reviewed, then the database queries can be executed.
444
00:28:55.880 --> 00:28:56.539
So, basically,
445
00:28:57.240 --> 00:28:59.980
code reviews are mandatory, especially for infrastructure
446
00:29:01.755 --> 00:29:02.255
changes,
447
00:29:02.794 --> 00:29:03.294
and,
448
00:29:04.235 --> 00:29:06.815
everything should be recorded. That's the first
449
00:29:07.515 --> 00:29:09.390
maybe okay. That's maybe the first thing.
450
00:29:10.510 --> 00:29:10.590
And,
451
00:29:11.390 --> 00:29:16.050
we, by the law, we must store data for 5 to 10 years, and,
452
00:29:17.230 --> 00:29:17.890
we have
453
00:29:18.785 --> 00:29:20.085
data retention policies
454
00:29:21.185 --> 00:29:22.485
defined and maintained
455
00:29:23.185 --> 00:29:28.580
on a permanent basis. So people aware that they cannot delete any financial documents whatsoever.
456
00:29:29.360 --> 00:29:29.680
If,
457
00:29:30.160 --> 00:29:31.380
even if they see,
458
00:29:31.760 --> 00:29:37.780
the company that is not touching the data already for a couple years and they say, okay. Well, they have, like,
459
00:29:38.085 --> 00:29:41.465
1, 000, 000, 000 of invoices there. Maybe you can delete something.
460
00:29:41.765 --> 00:29:43.385
They know they cannot because,
461
00:29:44.645 --> 00:29:50.920
maybe tomorrow someone will come and ask the the copy of the invoice that was issued, like, 4 years ago.
462
00:29:51.460 --> 00:29:51.960
And,
463
00:29:52.500 --> 00:29:53.000
oops,
464
00:29:53.700 --> 00:29:57.240
we're gonna have problems if if this data will get get deleted.
465
00:29:57.815 --> 00:29:58.315
And,
466
00:29:58.934 --> 00:30:00.075
yes. So that's
467
00:30:00.375 --> 00:30:01.414
3 major things.
468
00:30:01.815 --> 00:30:02.955
Access is restricted.
469
00:30:03.895 --> 00:30:07.680
Everything we need to do, in terms of and data management
470
00:30:08.060 --> 00:30:12.800
should be recorded and reviewed, and, all data records should be stored for years.
471
00:30:14.065 --> 00:30:14.565
Another
472
00:30:15.024 --> 00:30:19.365
aspect of that governance challenge is also on the side of discovery,
473
00:30:20.225 --> 00:30:25.330
because it's hard to do anything with the data if you don't know that you have it. And then once somebody,
474
00:30:25.810 --> 00:30:28.470
finds a particular dataset that they want to work with,
475
00:30:29.010 --> 00:30:35.044
I'm wondering what the process looks like for being able to request and gain access to that data
476
00:30:35.585 --> 00:30:44.460
and some of the audit controls that you have around how to ensure that they are doing what they say they were going to do and that the, access is managed appropriately?
477
00:30:45.640 --> 00:30:46.700
There is a bureaucracy,
478
00:30:47.080 --> 00:30:49.875
and we have we have to we have to
479
00:30:50.255 --> 00:30:53.395
enforce the bureaucracy on employees because all the certifications
480
00:30:54.175 --> 00:30:59.230
and compliance. So if you need to get access to data, you have to explain to
481
00:30:59.610 --> 00:31:04.190
dev ops team and to me, why do you need this and what exactly do
482
00:31:04.635 --> 00:31:06.735
you want to get from the database.
483
00:31:07.435 --> 00:31:08.975
And you have to also specify,
484
00:31:10.475 --> 00:31:11.915
the time span you want,
485
00:31:12.475 --> 00:31:14.630
have access to. Maybe have to specify,
486
00:31:15.490 --> 00:31:17.429
IDs of the customers of users
487
00:31:17.970 --> 00:31:19.429
you want to see in the database.
488
00:31:19.809 --> 00:31:21.410
And after that and after we,
489
00:31:22.215 --> 00:31:26.475
me and the engineering manager and DevOps team came up with a conclusion that,
490
00:31:27.095 --> 00:31:30.075
yes, you indeed can have access to that data.
491
00:31:30.450 --> 00:31:31.750
We'll manually execute
492
00:31:32.450 --> 00:31:39.270
the database queries and prepare the dump and hand this over to to 1 of the engineers and analysts. Obviously,
493
00:31:39.645 --> 00:31:44.145
all the critical information will be removed from that dump. So, let's say,
494
00:31:45.885 --> 00:31:47.160
invoice, well,
495
00:31:47.700 --> 00:32:00.825
IDs of the bank accounts or personal IDs will be completely removed. If the data is needed on a permanent basis, so for instance, you're working on a new data model, then we have to prepare something for the snowflakes. So we have to create
496
00:32:02.804 --> 00:32:04.940
a a data expert process that will be,
497
00:32:05.580 --> 00:32:10.399
sweeping off all the critical information and automatically pouring this data to to Snowflake.
498
00:32:11.100 --> 00:32:11.419
But,
499
00:32:11.980 --> 00:32:15.295
before we're doing this, we have to understand that we'll be,
500
00:32:16.155 --> 00:32:17.455
we'll be dealing with
501
00:32:17.835 --> 00:32:24.040
some portions of data on a permanent basis. Otherwise, we just do a manual dump and then give it to engineer,
502
00:32:24.500 --> 00:32:27.400
making sure that there is no critical data inside of this.
503
00:32:29.060 --> 00:32:32.360
In your work in this space and working with customers
504
00:32:32.755 --> 00:32:42.390
to enable them to build financial applications, what are some of the most interesting or innovative or unexpected approaches that you've seen to data management in this Fintech sector?
505
00:32:43.170 --> 00:32:46.690
In Fintech sector in general, I was surprised that,
506
00:32:47.170 --> 00:32:48.630
some companies, they're building
507
00:32:49.085 --> 00:32:51.745
quite sophisticated proxies for database servers.
508
00:32:52.045 --> 00:32:55.425
And as proxies, they're able to identify on the fly that
509
00:32:56.330 --> 00:33:00.830
direct credit card numbers stored in database, and as proxies will automatically
510
00:33:01.130 --> 00:33:05.230
remove all the critical information right from your database queries. So
511
00:33:05.885 --> 00:33:14.225
you won't be able to get the date access to that data, what's the error, and you don't have to write any configuration files. So obviously, the proxy servers,
512
00:33:14.559 --> 00:33:15.779
when you're writing this,
513
00:33:16.399 --> 00:33:21.759
config saying, okay. There is a column called card number. Please remove all the day,
514
00:33:22.955 --> 00:33:25.294
all the digits from there and just leave
515
00:33:25.595 --> 00:33:26.575
4 last digits.
516
00:33:26.875 --> 00:33:32.414
But you have to configure this this rules manually, and it's extremely annoying. So I was surprised to see
517
00:33:32.750 --> 00:33:43.885
that, machine learning can identify quite well all these patterns in database columns, and everything was and and everything will be stripped away on the fly. That's that's a game changer because,
518
00:33:45.164 --> 00:33:52.880
preparing data for, analytics analytics or for product managers is extremely annoying process, but we to deal with this anyways.
519
00:33:54.059 --> 00:34:04.065
And in your own experience of working in this space and building a company focused on Fintech, what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?
520
00:34:04.924 --> 00:34:09.859
Definitely had a challenge this year. It's not directly related to Fintech,
521
00:34:10.240 --> 00:34:14.260
but it's related to API First Business and the API First Company.
522
00:34:15.815 --> 00:34:17.435
It was surprising to me
523
00:34:18.055 --> 00:34:20.635
that dealing with multiple API versions,
524
00:34:21.335 --> 00:34:22.555
on a massive scale
525
00:34:22.855 --> 00:34:24.155
is pretty much tough,
526
00:34:25.000 --> 00:34:25.500
because
527
00:34:25.880 --> 00:34:34.300
we not only have to maintain multiple versions of APIs, but we also have different versions of data created by different API calls.
528
00:34:34.825 --> 00:34:37.085
And managing all these things,
529
00:34:37.705 --> 00:34:40.285
inside of application was indeed very challenging.
530
00:34:40.984 --> 00:34:41.484
So
531
00:34:42.340 --> 00:34:44.280
we identified 3 major strategies.
532
00:34:44.660 --> 00:34:47.160
And, first of all, we were thinking about
533
00:34:47.620 --> 00:34:49.320
store storing and managing,
534
00:34:49.700 --> 00:34:53.005
data in different Kubernetes clusters with different versions of APIs.
535
00:34:53.385 --> 00:34:56.845
It's obviously it's pretty tough approach because
536
00:34:57.305 --> 00:35:07.295
we have to maintain different branches of the code. And whenever you need to patch 1 cluster, you have to cherry pick your commits from the repository and have to somehow patch other clusters too.
537
00:35:07.855 --> 00:35:14.994
But the same time, you're still going to end up with multiple versions of data created from multiple cluster inside the database,
538
00:35:15.454 --> 00:35:17.000
which is which is nightmare.
539
00:35:17.780 --> 00:35:20.360
There is a second approach that is commonly used,
540
00:35:20.820 --> 00:35:23.640
in software companies. You just create little folders,
541
00:35:24.424 --> 00:35:35.070
obviously, different versions of your code. So let's say you have version 1, and now we have to create version 2. So you basically copy paste version 1 to folder called version 2, make changes there.
542
00:35:35.610 --> 00:35:41.605
But this code works with the same database, but same time well, looks like a very simple approach, but,
543
00:35:42.385 --> 00:35:47.500
if you're dealing with, like, a 10, 15 different versions of APIs and update that, that's
544
00:35:48.460 --> 00:35:49.360
extremely challenging.
545
00:35:49.820 --> 00:35:53.680
And we're API first company, and we're dealing with with Fintech. So we have,
546
00:35:54.060 --> 00:36:07.359
some our customers using our APIs for years. So we just simply cannot kill old versions because there is always company using this, and we just cannot send them a letter saying, yes. We're going to switch out this version completely,
547
00:36:07.819 --> 00:36:08.780
something from,
548
00:36:09.420 --> 00:36:20.055
January 1st. Well, please switch. Well, we can't because they pay money for us. So we have to support this more multiple versions, and that was the approach we initially took. So we created these multiple folders,
549
00:36:21.010 --> 00:36:22.869
and, we had all these problems.
550
00:36:23.250 --> 00:36:24.390
And 1 of the engineers,
551
00:36:25.730 --> 00:36:26.470
he created
552
00:36:26.770 --> 00:36:27.589
a framework
553
00:36:27.890 --> 00:36:29.030
for API versioning.
554
00:36:30.125 --> 00:36:39.550
The philosophy behind this solution is quite simple. So we always have last version of data available in database and last version of the business logic.
555
00:36:40.110 --> 00:36:44.850
Now if you have to serve data to someone who is using previous versions of APIs,
556
00:36:46.110 --> 00:36:47.490
this framework basically
557
00:36:47.870 --> 00:36:48.370
downgrades
558
00:36:49.655 --> 00:36:51.115
the data with the presentation
559
00:36:51.975 --> 00:36:56.795
to, to the requested version. So let's say we have 5 versions in total. So last version's
560
00:36:57.250 --> 00:37:01.110
version number 5. So all data and database and all the data warehouse
561
00:37:01.490 --> 00:37:04.470
is stored according to v 5 specifications.
562
00:37:05.055 --> 00:37:06.115
But if you have customer
563
00:37:06.415 --> 00:37:08.435
using v 4, then we basically,
564
00:37:09.455 --> 00:37:15.380
downgrade their v 5 back to v 4. It's like a, database migration process, but done backwards.
565
00:37:16.080 --> 00:37:16.580
And
566
00:37:17.760 --> 00:37:22.420
turns out that it it's extremely efficient process. The the number of lines of codes
567
00:37:23.295 --> 00:37:28.115
dropped dramatically. We removed, like, a hundreds and thousands of lines of code that
568
00:37:28.415 --> 00:37:32.275
was so hard to maintain, and we ended up with this solution. So we always
569
00:37:32.640 --> 00:37:36.420
store the latest version of data in our data system,
570
00:37:37.040 --> 00:37:41.695
designed according to latest version of text specs we have, and we just migrate
571
00:37:42.155 --> 00:37:42.975
all the data
572
00:37:43.915 --> 00:37:44.415
representations
573
00:37:44.795 --> 00:37:47.295
backwards to to the requested version.
574
00:37:47.755 --> 00:37:49.935
API versioning is a hard challenge
575
00:37:50.260 --> 00:37:58.920
no matter what space you're in, but having to maintain it for a indeterminate amount of time definitely adds a significant amount of burden and probably
576
00:37:59.575 --> 00:38:03.115
technical debt to ensure that you're able to maintain that backwards compatibility.
577
00:38:03.734 --> 00:38:04.234
Sure.
578
00:38:05.255 --> 00:38:17.414
Yeah. That that's definitely a very interesting solution that you found to that, being able to maintain the latest representation of the data, but then being able to translate it backwards. So that's that's interesting. And,
579
00:38:18.355 --> 00:38:19.815
so you mentioned that you
580
00:38:20.434 --> 00:38:25.335
keep the latest in the prior version. Do you have to because of the fact that you have people using
581
00:38:25.680 --> 00:38:27.940
some indeterminate version of the API for
582
00:38:28.480 --> 00:38:29.140
in perpetuity,
583
00:38:29.440 --> 00:38:35.060
do you have to maintain the, you know, multiple versions and multiple downgrade steps for a particular data representation?
584
00:38:36.964 --> 00:38:37.464
Yeah.
585
00:38:37.924 --> 00:38:38.424
Basically,
586
00:38:39.125 --> 00:38:45.440
it's it's it's it's pretty it's pretty hard to to jump from v 5 back to v 1. So probably,
587
00:38:46.460 --> 00:38:49.520
it's a good option to to do a couple of migrations in the middle.
588
00:38:49.900 --> 00:38:50.275
And
589
00:38:50.994 --> 00:38:56.934
that also is very helpful when you're doing debugging because you can see all these changes between different versions.
590
00:38:57.539 --> 00:39:01.559
Yes. It's still pretty annoying process because you have to migrate data representations.
591
00:39:02.420 --> 00:39:02.920
And
592
00:39:03.779 --> 00:39:06.680
as you continue to build and iterate
593
00:39:07.035 --> 00:39:26.015
on the product that you're building at Monite and invest in this Fintech sector and enabling other businesses to build financial applications on top of your platform, what are some of the things you have planned for the near to medium term as far as your data capabilities or your platform architecture at Monite or some of the problems and projects you're needed to dig into?
594
00:39:26.635 --> 00:39:27.295
We have,
595
00:39:29.355 --> 00:39:34.349
more like a long term strategic plan because we're dealing with quite a lot of transaction information
596
00:39:34.650 --> 00:39:36.750
coming from SMEs in different countries.
597
00:39:37.609 --> 00:39:38.990
We came up with our
598
00:39:39.530 --> 00:39:45.215
ultimate goal and with our North Star. We're eventually planning to to keep, to kill all,
599
00:39:46.395 --> 00:39:54.119
financial documents and OCR in fintech space because there is a there is a swift product of financial transactions,
600
00:39:55.460 --> 00:39:57.559
made between different banks. And
601
00:39:59.005 --> 00:40:00.145
before that, people
602
00:40:00.525 --> 00:40:03.744
used used telegraph in order to make money transfers.
603
00:40:04.445 --> 00:40:04.945
But,
604
00:40:05.860 --> 00:40:13.145
right now, we live in 21st century, and people still print invoicing and paper and delivering these invoices to each other via physical mail.
605
00:40:13.465 --> 00:40:22.210
And, our plan is to completely kill that industry. Obviously, it's kind of ambitious, but we're gonna take baby steps towards that goal. And probably it's gonna take,
606
00:40:22.910 --> 00:40:24.130
5 years at least.
607
00:40:24.670 --> 00:40:28.130
But the idea is to completely kill all these PDF documents,
608
00:40:28.505 --> 00:40:30.605
scans, and replace them with machine readable,
609
00:40:31.305 --> 00:40:35.724
messages. And, these messages should have very strict structure inside.
610
00:40:37.230 --> 00:40:39.410
So instead of also wiring your
611
00:40:40.110 --> 00:40:42.030
PDF files, you'll get access,
612
00:40:42.670 --> 00:40:44.610
to all your financial records instantly,
613
00:40:44.994 --> 00:40:46.515
and have to manually adjust,
614
00:40:46.835 --> 00:40:48.775
and verify your financial records.
615
00:40:49.315 --> 00:40:51.015
I guess it's something that's,
616
00:40:52.195 --> 00:40:56.869
will probably eliminate thousands of the jobs because someone has still has to scan
617
00:40:57.170 --> 00:41:01.750
and print these papers and deliver. But I guess it's a good thing to do, because,
618
00:41:02.734 --> 00:41:09.875
well, yes, 21st century. We definitely can do better than a bunch of PDL files and paper envelopes developed,
619
00:41:10.255 --> 00:41:11.075
by a mailman.
620
00:41:11.460 --> 00:41:12.920
That's that's in our store.
621
00:41:13.460 --> 00:41:15.940
But on the way to that, to that goal,
622
00:41:16.660 --> 00:41:19.560
we're gonna we're gonna keep expanding in different markets.
623
00:41:20.224 --> 00:41:24.305
It's pretty challenging process because we're infrastructure company, and we have,
624
00:41:24.785 --> 00:41:28.565
spikes of traffic. So when we onboarding on onboard new customer,
625
00:41:29.650 --> 00:41:31.590
you may, start serving,
626
00:41:31.890 --> 00:41:34.470
let's say, a 100 k new users overnight.
627
00:41:34.850 --> 00:41:34.930
And,
628
00:41:35.970 --> 00:41:42.214
usually, when you're doing, b to c software, you don't see this kind of, traffic spikes,
629
00:41:42.675 --> 00:41:43.895
quite often. Usually,
630
00:41:44.435 --> 00:41:46.295
you see a number of users,
631
00:41:46.675 --> 00:41:47.734
gradually increasing,
632
00:41:48.115 --> 00:41:54.040
over time. But if you have platform and, your customer just start importing their users,
633
00:41:55.380 --> 00:41:56.200
to infrastructure,
634
00:41:56.580 --> 00:41:59.045
then, yes, there are spikes. And dealing with the spikes
635
00:41:59.444 --> 00:42:00.405
in something that's,
636
00:42:01.525 --> 00:42:04.505
is is it it's it's pretty hard process because,
637
00:42:04.885 --> 00:42:06.585
yes, we have plans for everything.
638
00:42:06.964 --> 00:42:07.710
We we have,
639
00:42:08.750 --> 00:42:10.369
all the all the infrastructure,
640
00:42:11.630 --> 00:42:17.010
challenges modeled, and we're prepared for everything. But reality is, however, a bit different. And,
641
00:42:18.954 --> 00:42:23.615
yes, that that's a that's a that's a long process of our system evolution
642
00:42:23.994 --> 00:42:25.135
towards that goal.
643
00:42:26.490 --> 00:42:38.565
Are there any other aspects of the work that you're doing at Moneight or the overall problem space of data management in a Fintech in a Fintech context that we didn't discuss yet that you would like to cover before we close out the show?
644
00:42:39.105 --> 00:42:42.339
Yeah. There is 1 interesting topic. So currently,
645
00:42:42.800 --> 00:42:43.280
all,
646
00:42:43.760 --> 00:42:44.980
pretty much all garments,
647
00:42:45.359 --> 00:42:48.099
across the globe, they they're trying to introduce local
648
00:42:48.400 --> 00:42:50.180
standards for financial documents.
649
00:42:50.605 --> 00:42:52.704
And seems like a nightmare because
650
00:42:53.164 --> 00:42:59.025
even in European Union, now we have Italian invoicing standard, Portuguese invoicing standard.
651
00:42:59.400 --> 00:43:00.299
We have German,
652
00:43:00.680 --> 00:43:02.220
standard for financial documentation.
653
00:43:03.000 --> 00:43:03.819
And, basically,
654
00:43:04.280 --> 00:43:07.180
we have the zoo of data formats and standards,
655
00:43:07.645 --> 00:43:12.065
and some of the standards are not even to even have any explanation
656
00:43:12.365 --> 00:43:18.320
in English how it works. So if you want to, get connected with Italian invoice and you have to circle through,
657
00:43:20.300 --> 00:43:21.040
2 or 300
658
00:43:21.420 --> 00:43:28.255
page long document written in Italian language explaining how the scene wasn't works, and after you have to give a call to,
659
00:43:29.035 --> 00:43:33.694
Italian authorities and say, look, like, you would have a plan to get connected with your
660
00:43:34.400 --> 00:43:36.020
documents and exchange gateway.
661
00:43:36.400 --> 00:43:37.220
So, basically,
662
00:43:38.400 --> 00:43:40.900
that's a big night nightmare that is coming
663
00:43:42.160 --> 00:43:43.540
on our heads. And,
664
00:43:44.555 --> 00:43:48.255
I I hope, like, in the 5 to 10 years, people,
665
00:43:49.195 --> 00:43:51.615
will abandon this idea because right now,
666
00:43:52.250 --> 00:43:52.750
it
667
00:43:53.210 --> 00:44:01.375
just multiplies the problems. And instead of just dealing with PDF files and financial records coming from different banking systems, we now have to deal with,
668
00:44:02.335 --> 00:44:03.795
hundreds of different XMLs,
669
00:44:04.095 --> 00:44:07.635
JSONs, or binary formats. And so it's it's it's
670
00:44:08.415 --> 00:44:11.315
it's just for it's just, a very big challenge.
671
00:44:12.880 --> 00:44:13.860
Yeah. The
672
00:44:14.400 --> 00:44:15.780
is there at least any
673
00:44:16.400 --> 00:44:18.875
sort of common subset of information is
674
00:44:19.895 --> 00:44:22.955
there at least any sort of common subset of information across those standards, or is it every country has decided on their own bespoke
675
00:44:23.335 --> 00:44:27.040
format that you they are forcing people to comply with and you have to do
676
00:44:27.520 --> 00:44:29.620
a specific implementation for each 1?
677
00:44:31.680 --> 00:44:35.060
So there is no definite single standard across different countries,
678
00:44:35.360 --> 00:44:35.825
and
679
00:44:38.065 --> 00:44:38.724
I guess,
680
00:44:40.625 --> 00:44:45.365
governments don't have any intentions even to start discussing these topics because everyone
681
00:44:46.050 --> 00:44:49.190
says that, okay. Well, it's time to introduce our own standard
682
00:44:49.650 --> 00:44:49.890
and,
683
00:44:50.850 --> 00:44:54.395
get some extra money from the taxpayers because the invoicing,
684
00:44:54.775 --> 00:44:58.875
will get give us better access to data and, hence, better access to taxes. So
685
00:44:59.175 --> 00:45:00.535
everybody is excited by,
686
00:45:01.930 --> 00:45:02.830
everybody is
687
00:45:03.210 --> 00:45:06.350
excited excited by the fact they can get extra money from the taxpayers,
688
00:45:06.730 --> 00:45:10.510
but nobody is is thinking about the consequences. So that's a problem.
689
00:45:11.095 --> 00:45:26.770
Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you and your team are doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today.
690
00:45:28.425 --> 00:45:29.484
Well, definitely,
691
00:45:30.265 --> 00:45:32.925
in the Fintech space, we have to build
692
00:45:33.545 --> 00:45:34.045
manually
693
00:45:34.425 --> 00:45:38.270
all this data management solution. As I said before, there are lots of restrictions.
694
00:45:39.210 --> 00:45:40.350
There are lots of,
695
00:45:41.130 --> 00:45:44.270
rules that enforced by, local laws and regulations.
696
00:45:45.050 --> 00:45:45.550
And
697
00:45:46.075 --> 00:45:52.974
that will be absolutely awesome to see a database or data storage solution that will take care about all these things,
698
00:45:53.755 --> 00:45:54.255
because,
699
00:45:55.360 --> 00:45:57.780
roughly saying we spend, like, a 25%
700
00:45:58.080 --> 00:46:00.400
of our tech team budget on on,
701
00:46:00.800 --> 00:46:02.340
dealing with this and
702
00:46:03.125 --> 00:46:03.944
having database that
703
00:46:04.405 --> 00:46:05.224
is already
704
00:46:05.924 --> 00:46:06.424
prepared
705
00:46:06.724 --> 00:46:08.505
to face all these Fintech challenges
706
00:46:08.805 --> 00:46:13.680
and that is already combined with all these laws and regulations, that will be a game changer.
707
00:46:14.700 --> 00:46:17.360
I guess it's not hard to build this
708
00:46:18.704 --> 00:46:20.405
technology from technical perspective.
709
00:46:20.944 --> 00:46:23.125
But from the compliance perspective,
710
00:46:23.744 --> 00:46:27.444
it's complete nightmare. And probably in order to achieve that,
711
00:46:28.029 --> 00:46:31.809
someone needs to have, like, a 10 skilled software engineers and 100
712
00:46:32.349 --> 00:46:38.525
compliance managers and and lawyers and product managers in order just to explain how it's supposed to work.
713
00:46:39.225 --> 00:46:39.725
It's
714
00:46:40.105 --> 00:46:44.045
I I hope someone someone will create that solution in the future because,
715
00:46:44.460 --> 00:46:48.800
it's just so painful to to deal with this all these all these things manually.
716
00:46:49.500 --> 00:47:08.450
Alright. Well, thank you very much for taking the time today to join me and share your experience and perspectives on data management in Fintech and the work that you're doing at Moneight. It's definitely a very interesting and important problem domain, so I appreciate all of the time and energy that you and your team are putting into that, and I hope you enjoy the rest of your day.
717
00:47:15.710 --> 00:47:16.609
Thank you for listening.
718
00:47:16.910 --> 00:47:19.490
Don't forget to check out our other shows, podcast.init,
719
00:47:19.950 --> 00:47:25.945
which covers the Python language, its community, and the innovative ways it is being used, and the Machine Learning Podcast,
720
00:47:26.405 --> 00:47:30.825
which helps you go from idea to production with machine learning. Visit the site at dataengineeringpodcast.com
721
00:47:32.570 --> 00:47:41.555
Subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a product from the show, then tell us about it. Email hosts at dataengineeringpodcast.com
722
00:47:42.975 --> 00:47:49.090
with your story. And to help other people find the show, please leave a review on Apple Podcasts and just tell your friends and coworkers.