درباره این اپیزود
Brian Green, software architect at SnapLogic, joins Dominic and Guy to explore the fascinating convergence of no-code, low-code, and traditional software development strategies. Discover why even the most user-friendly platforms require a solid grasp of software principles to truly excel, especially in large-scale operations. Brian sheds light on how traditional concepts like configuration management, environmental separation, and documentation are essential in optimizing no-code development environments. With a focus on the balance between ease-of-use and the rigor of traditional development practices, this episode promises to equip you with the insights needed to navigate the challenges of scaling in the digital age.
Join us as we uncover how traditional software techniques, such as linting, can revolutionize SnapLogic pipelines for enhanced robustness and efficiency. Learn about the seamless integration of these techniques with source control and how a center of excellence can bridge the gap between traditional and citizen developers. We delve into the potential of low-code/no-code platforms when the right levels of abstraction are understood. From generating test data with tools like SnapGPT to balancing standard engineering practices with unique platform benefits, this episode is packed with insights to elevate your enterprise strategies.
یادداشت ها را نشان دهید 🔗
رونوشت 🔗
00:00:00.119 --> 00:00:09.912
Hello and welcome back to the Enterprise Alchemists, a podcast by SnapLogic that is all for enterprise architects and people who want to be enterprise architects.
00:00:09.912 --> 00:00:16.349
I'm your host, ominic Wellington, and also joined by my colleague, guy, the other host.
00:00:16.349 --> 00:00:17.170
Hi, guy,
00:00:17.170 --> 00:00:18.794
reetings all
00:00:18.794 --> 00:00:28.035
, nd this week, continuing a minor theme that we had from the last season, we're joined by someone who is a different sort of architect.
00:00:28.035 --> 00:00:31.248
So Brian Green is a colleague of ours at SnapLogic.
00:00:31.248 --> 00:00:34.871
His title is architect, but he's not one of our merry band.
00:00:34.871 --> 00:00:38.526
Brian, thank you for joining us and could you please explain?
00:00:40.381 --> 00:00:42.188
Hi, I'm Brian Green.
00:00:42.188 --> 00:00:50.799
I have the ambiguous title at SnapLogic, just architect, and really it's the internal side of things and the product.
00:00:50.799 --> 00:01:24.105
So I started my career doing web apps, early infrastructure, and moved into data and analytics and then moved into moving these things analytics so and then moved into moving these things, and so you know, having moved app data and having moved data data, um, snaplogic has been a nice, a nice fit in that you know we do all those things, um, so I work on the inside of the product, um, trying to optimize it and look at what is the next generation of things that we're going to do internally to drive new capabilities for the customer.
00:01:25.325 --> 00:01:25.667
Excellent.
00:01:25.667 --> 00:01:44.745
So that was actually what we wanted to talk about for the next little while, because one of the conversations that I certainly and I believe, guy, it's the same for you conversation that we frequently end up having with our counterparts on the customer side is, you know, the SnapLogic thing.
00:01:44.745 --> 00:02:04.852
It might not look like your traditional text-oriented programming tool, but actually there's a whole lot of programming best practices let's say that come from that world that you should keep in mind when you're developing and when you're managing, especially a larger amount of SnapLogic pipelines.
00:02:04.852 --> 00:02:15.532
And so he wants to talk a little bit about how you could apply some of those traditional software development best practices to the sorts of no-code development that goes on inside of SnapLogic.
00:02:15.532 --> 00:02:23.229
So we had a great beginning of a conversation and we decided, no, no, no wait, this needs to be a podcast.
00:02:23.229 --> 00:02:24.665
We need to share this with the world.
00:02:24.665 --> 00:02:33.664
So do you have an introductory statement, Brian brian, that you want to lay out your position?
00:02:33.695 --> 00:02:34.094
rian man, I think you alluded to it reasonably well.
00:02:34.094 --> 00:02:38.512
You know this was my, you know my thought when I came to SnapLogic and things I was asking about, kind of on day one.
00:02:39.980 --> 00:02:49.085
A history of no-code, low-code tools does show that they're really they continue to just be abstractions over the computer.
00:02:49.085 --> 00:02:54.361
Uh, it's a visual abstraction, um, instead of a heavy text-based abstraction.
00:02:54.361 --> 00:02:54.723
You know it's.
00:02:54.723 --> 00:03:04.314
It can be a lot easier and more efficient to move a lot more data around, but it doesn't change some of the fundamentals about what we want to think about.
00:03:04.314 --> 00:03:13.114
You know, environmental separation, how do we do testing, how do we document and do lineage and think about our data and governance.
00:03:13.114 --> 00:03:23.264
You know these concerns don't go away and you know, I think with a lot of tools you see the same lifecycle, right, like you could name some.
00:03:23.264 --> 00:03:28.563
You could name some other ones that start with an S ironically, several of them, right.
00:03:29.406 --> 00:03:33.743
And you know you come in and you say, well, it's no code and it's very easy, and we get it moving.
00:03:34.365 --> 00:03:42.921
And then you know you get six months in or a year in and the scale has gone up and the criticality has gone up and all of a sudden you start to have challenges.
00:03:42.921 --> 00:03:49.606
They're subtle, weird things and a lot of people will ask themselves a lot of questions but why is this hard?
00:03:49.606 --> 00:03:51.967
Why isn't it working as well as we thought it would?
00:03:51.967 --> 00:04:06.921
And some of the answer is you're doing software work, but you didn't treat it like software, and that's okay in the beginning and when it's easy, but over time that's okay in the beginning and when it's easy, but over time that's challenging.
00:04:06.921 --> 00:04:12.204
And so you don't need compilers and green screens, right, we don't have to make this hard.
00:04:12.204 --> 00:04:27.612
But the lessons learned and the way we do things like configuration management, those are pretty universal and so I think, dragging them into, how do you function with a low code, no code tool at scale, getting the benefits of that?
00:04:27.612 --> 00:04:36.255
Yet Okay, really we are doing software right, and there's an admission there in the we are doing software that can be challenging sometimes.
00:04:36.915 --> 00:04:40.778
The first step is you have to admit to yourself that you're perpetrating some software here.
00:04:45.699 --> 00:04:47.204
I've had that conversation with several people in IT management.
00:04:47.204 --> 00:04:48.208
Right, like we just have to to be a little bit.
00:04:48.208 --> 00:04:48.569
This is software.
00:04:48.588 --> 00:04:59.423
It's easy and it's fast, but it is software I'll just add to that comment because, um and again I'll come from a different background of with, of integration, data management.
00:04:59.543 --> 00:05:14.915
But even with our larger clients, it's interesting that I've seen in the last 18 months there is a duality across some of our accounts about that approach of the.
00:05:14.915 --> 00:05:55.870
We started as low-cost citizen developer this is like the net new thing that doesn't need these concepts but then we're seeing accounts with hundreds of developers, thousands and even tens of thousands of pipelines, so obviously integration patterns and components and snaps and potentially custom functions, and not realizing at that point that and not realizing at that point that you that moment of realization that says you need intelligence automation, above and beyond what the UI is giving the end user.
00:05:55.870 --> 00:05:59.470
So completely agree with everything you're saying.
00:05:59.470 --> 00:06:14.882
But I have been finding it interesting that in our larger accounts it's not every single one of them has realized this and I think this is also a wave I'm starting to hear about across quite a lot of the medium to large enterprises we're talking to that.
00:06:14.882 --> 00:06:20.295
Sometimes they kind of miss the elephant in the room when they start talking about strategic management.
00:06:20.516 --> 00:06:32.247
Yeah, I agree, I'm seeing some of the same in one of my accounts has a couple of thousand monthly acts, active users in the platform developing pipelines, and this was something that's actually a slight aside.
00:06:32.247 --> 00:06:34.305
This was actually something that surprised me a little.
00:06:34.305 --> 00:06:42.211
Coming from outside the field, I had an expectation in the back of my mind that data integration would mostly be an occasional thing.
00:06:42.211 --> 00:07:02.141
You'd set up the pipeline and then the pipeline would run and you'd go do something else, and the rate of change inside these large environments and the never-ending flow of new bright ideas, of things we could do if we could only get access to that bit of information over in that other system, means that there is actually a pretty constant flow of developments.
00:07:02.141 --> 00:07:06.509
It's not at all a static environment, but I think that's the realization.
00:07:06.509 --> 00:07:30.745
Also many of those customers are having that they start off not thinking that they need to worry about you know which is the canonical pipeline, and all of them end up with weird names and things like that and suddenly it's two years later and the environment is not the way that they would design it if they knew everything at the time that they have since found out.
00:07:31.286 --> 00:07:41.946
So a lot of what this podcast is about is about trying to and indeed our whole team's practice is about trying to capture that information and help people avoid getting in trouble.
00:07:41.946 --> 00:07:49.673
Learning from other people's mistakes rather than from your own is generally a more pleasant experience, and so that's kind of where we are today.
00:07:49.673 --> 00:08:06.014
So, coming back to the nominal point, why don't we walk through some things, some techniques that come from the more traditional text-oriented development world and how we might apply them in the SnapLogic world?
00:08:06.014 --> 00:08:12.129
So one that Guy and I were actually just discussing offline was linting, for instance.
00:08:12.129 --> 00:08:19.307
This is something that's you know in the software development text-oriented, again, software development world.
00:08:19.307 --> 00:08:26.141
There's all this idea of being able to identify difficult, problematic, suspicious ways of doing things in your code.
00:08:26.141 --> 00:08:29.971
How would that apply to SnapLogic Pipeline, would you say?
00:08:29.971 --> 00:08:30.132
Brian?
00:08:30.639 --> 00:08:42.048
I mean, I think it's a similar idea, right, there are static patterns in combinations of snaps or snap configurations that you can analyze for Very.
00:08:42.107 --> 00:08:45.455
You know static patterns in any source code language.
00:08:45.455 --> 00:08:53.279
It's been fascinating to see linting for snap logic pipelines show up as snap logic pipelines.
00:08:53.279 --> 00:08:56.633
That was a thing that was an interesting surprise to me.
00:08:56.633 --> 00:09:20.904
But yeah, I think it's a similar concept and the emergence of that in the last I don't know couple of years in the platform does, uh, I guess a real signal that you know you've developed a bit of a programming language right, like if you can lint it and you know if you can do things with a language that cause problems such that you want to do static analysis to detect those.
00:09:20.904 --> 00:09:24.176
You know there's you've crossed a boundary into.
00:09:24.176 --> 00:09:40.289
Okay, you have produced something that looks like a programming language, right, because it's general purpose enough that you can say that there are things that are semantically correct that also you shouldn't do, right, which is why we do linting in, you know, in all languages.
00:09:40.289 --> 00:09:44.501
So it's an interesting technique and, yes, available.
00:09:44.783 --> 00:09:54.068
There's probably a convergence there that if the environment is complex enough to need linting, it's also complex enough to support linting, as you just said.
00:09:54.068 --> 00:10:02.706
You know you can do it as a pipeline in SnapLogic and now we have the ability to call that even from SnapGPT so that you can check your pipelines for best practices.
00:10:02.706 --> 00:10:21.129
So that's something that you might not think of if you don't come from a programming background, but it's definitely a thing that we're trying to do to make more visible so that people will think of checking and will have the advantage or the benefit of doing that.
00:10:21.480 --> 00:10:29.595
I'm really interesting to see how we get that linting capability integrated with source control right.
00:10:29.595 --> 00:10:32.409
I mean, that's where we usually see linting and controls.
00:10:32.409 --> 00:10:45.620
So I think, as we look at you know, talk to product management about how does this fit into the product, how do you get the benefit of these things from the software engineering without talking to people about the complexity of it?
00:10:45.620 --> 00:10:52.066
Right Like, there's an argument that says I shouldn't need to think about linting, right Like, as a user.
00:10:52.066 --> 00:10:53.489
Why doesn't the system do that for me?
00:10:53.489 --> 00:11:00.759
You know, and there's this back and forth iterative conversation, and so, you know, where does it fit into the integration with source control?
00:11:00.759 --> 00:11:04.024
Do I have to manually turn that on Right Like?
00:11:04.024 --> 00:11:09.714
If you're telling me that linting is the best practice, then shouldn't I just have to work to turn it off?
00:11:10.394 --> 00:11:13.818
I think there's a bigger you know bigger conversation about how to you know, how does it feature?
00:11:13.818 --> 00:11:20.826
You know, how does a traditional software engineering feature that developers are used to manually configure, like you know, thinking about?
00:11:20.826 --> 00:11:24.462
What are we learning from platform engineering and other cross disciplines?
00:11:24.462 --> 00:11:29.253
Why does the developer have to think about that right Like?
00:11:29.253 --> 00:11:34.870
From a developer experience standpoint, can't we just make that kind of more transparent?
00:11:35.111 --> 00:11:36.033
And I think we'll get there.
00:11:36.836 --> 00:12:48.570
I think, yeah, that's the realization that some of the longer standing customers that I work with some of them have a decade of experience with the tool, that sort of thing that they're getting to adding all sorts of checks as part of a multi-stage release and deployment process, and it's one of the big things that we talk about a lot on our team when we're advising SnapLogic users on setting up a center of excellence because, that center of excellence can be staffed by people who have this sort of experience to think about things like CICD, and that is what enables you to have the true citizen developers, people who come from a domain background, domain specialist background, so that they can work on that side of things with their closer understanding of the domain of the problem that they're actually trying to solve, advised and supported by things that are being done at the center of excellence level by people who have that sort of background yeah, absolutely, and I think there's an, there's another, I'm not going to say challenge, but I think the other thing I'm seeing across this comment is, again, we obviously snap logic is a low code, no code environment.
00:12:50.734 --> 00:13:02.753
I think one of the challenges and opportunities for us to consider this moving forward is we also have a lot of environments where you, you do have very different classes of user.
00:13:02.753 --> 00:13:08.172
So we do have, they are obviously technically orientated, but they, they themselves, are not developers.
00:13:08.172 --> 00:13:17.217
They're not, they don't live within a defined software lifecycle within their job, they are much more point-orientated.
00:13:17.217 --> 00:13:37.976
And then, combining that with, we have integration developers that I would say, historically at least, have been closer to the traditional developers because of the nature of the patterns they produced, whereas, again, I see a lot of data users can quite often be a little bit more less engineering, more point solution orientated.
00:13:38.597 --> 00:13:47.892
I think you raise an interesting question about when and should you be giving this information to the end user, if you do or don't understand, if they understand the outcome.
00:13:47.892 --> 00:14:01.155
Um, in several of the accounts I'm working with, they are using this within the operational life cycle of the pipeline migration rather than the developer life cycle, if that makes sense.
00:14:01.155 --> 00:14:13.991
But yes, I mean obviously the panacea would be we have something that you literally could, you know the dream would be it's so rich that you just it's on and it almost raises its head only when things are going to go badly.
00:14:13.991 --> 00:14:31.352
But that's obviously more of a vision today than where we're at with the successful early days that we're at with the current offerings yeah, there's the old joke that the perfect software user interface should just have a single button labeled do what I mean you know, I posted on linkedin somewhat recently.
00:14:32.193 --> 00:14:33.534
Uh know, I know someone.
00:14:33.534 --> 00:14:35.897
They went to a meeting we met with the business.
00:14:35.897 --> 00:14:54.950
The customer was very savvy and after about 30 minutes of discussion they said, listen, I need you to just make the software flexible so it can do what I need need.
00:14:54.950 --> 00:14:56.352
And I mean this was the level of sternness, right.
00:14:56.352 --> 00:15:00.842
I mean that like she pounded her hand on the tape right, like she was looking us in the eye and like, okay, and you know, we confirmed it.
00:15:00.842 --> 00:15:04.918
Oh, that's the requirement make it flexible so it does what you need.
00:15:04.918 --> 00:15:06.121
Yes, you finally get it.
00:15:06.121 --> 00:15:07.591
Okay, we got it.
00:15:07.591 --> 00:15:14.585
And then you know the meeting was over and I mean we didn't got it right, like we kind of got it.
00:15:14.585 --> 00:15:24.241
You know it was data analytics and dashboarding kinds of things, right, and you get into the what's the requirement for the infinitely flexible dashboard?
00:15:24.241 --> 00:15:26.256
It's that it does what I think.
00:15:26.710 --> 00:15:29.416
It shows me what I need to know when I need to know it indeed.
00:15:29.456 --> 00:15:30.778
And you know you wish your integration.
00:15:30.778 --> 00:15:39.057
I integration, I mean I still think there's like a, as we look at Chad GPT, I still think there's vision there, particularly with integration around.
00:15:39.057 --> 00:15:49.356
Hey, I've got this kind of system and this kind of system connect them together, right Like I still think that there's an easy button there or easier button there.
00:15:49.958 --> 00:16:08.884
Indeed, but while we still have to operate in this broken world, one of the things that we do have to manage, if you want to set up a full-on continuous integration, continuous deployment model is you have to think about multiple environments that you work with.
00:16:08.884 --> 00:16:14.011
What do you think people are not doing in SnapLogic that they ought to be doing by and large?
00:16:14.011 --> 00:16:17.782
What are the sorts of things that you think should be more prevalent out there?
00:16:18.929 --> 00:16:20.616
I actually think this is a thing we do easy.
00:16:20.616 --> 00:16:36.634
This one, like the use of accounts to abstract the connection details away from a pipeline, is a very straightforward part of the tool and I think anybody almost anybody grasps that abstraction very quickly.
00:16:36.634 --> 00:16:45.003
Right, and when I think about the, you know the oldest abstraction for this kind of thing, thinking about this podcast before we got on.
00:16:45.003 --> 00:16:56.817
You know, I started writing software 20, over 20 years ago and in your java application when you wanted a connection to a database, you'd ask for it by name, right like.
00:16:56.817 --> 00:17:02.360
You wouldn't say, hey, I want, you know, I know the name of the database server and the credentials and all this other madness.
00:17:02.360 --> 00:17:04.569
Right like, no, you're a piece of software.
00:17:04.569 --> 00:17:06.696
Why would you even have any of that information?
00:17:06.696 --> 00:17:14.942
You would ask the thing that has the server connections to give you one and you'd say, hey, I it by name and it would give it back to you by name.
00:17:15.490 --> 00:17:21.980
And then later, when your software ran in a different environment, you know, as long as the administrator had set that up, that just worked.
00:17:21.980 --> 00:17:27.653
And you know I've seen lots of software that made that problematic and over the years.
00:17:27.653 --> 00:17:40.758
But you know this is a thing SnapLogic does very well right, like, and I think most people building pipelines and sort of configuring things get to very quickly understanding how that works and you know you can do quite some magic with it if you want.
00:17:40.758 --> 00:17:45.359
So I don't know, I don't know that this is a very big deficiency I see in pipelines.
00:17:45.359 --> 00:17:50.059
I think we make it easy enough to do it right.
00:17:50.930 --> 00:17:51.974
Exactly, exactly.
00:17:51.974 --> 00:17:55.203
Yeah, because there is always that trade-off.
00:17:55.203 --> 00:17:58.056
You have a lot of capabilities that you can offer.
00:17:58.056 --> 00:18:14.122
You can be flexible enough to do what the user wants, but you also there's a sort of magic about when to disclose that, when to force the user to think about something and when to let them get to it in their own time, when they have the need.
00:18:14.122 --> 00:18:16.056
And that's the trick.
00:18:16.056 --> 00:18:24.118
In this case, yeah, the multiple environment, multiple user parameter integration is hitting the correct balance.
00:18:24.118 --> 00:18:28.079
Now I'm wondering what else we should be doing even more of.
00:18:28.549 --> 00:18:32.257
I mean, one of the things that I think is challenging in this space is lineage.
00:18:32.257 --> 00:18:36.200
You know, lineage is something we've been working on for a while.
00:18:36.200 --> 00:18:37.736
We've been talking about it.
00:18:37.736 --> 00:18:41.932
There's some early previews out there In integration and environment.
00:18:41.932 --> 00:19:10.943
Lineage and observability both ideas that we kind of can take out of software they're really key and I think they've been challenging for our customer and for lots of customers, right Like, lineage is a huge subject across data integration platforms and if you look at how a pipeline is built, if there's any level of dynamic SQL in it or dynamic configuration, you actually can't understand the lineage for the pipeline until it runs.
00:19:11.469 --> 00:19:11.589
Yeah.
00:19:11.871 --> 00:19:15.300
You know true design time lineage.
00:19:15.300 --> 00:19:23.525
If you have an engine that gives you that, I make the argument that it's not flexible enough to be super useful.
00:19:23.525 --> 00:19:26.198
Right, like you'd be mad at it if it could do it.
00:19:26.198 --> 00:19:40.623
And so the flip is and this is fun to talk to as we've been designing this feature with customers right, because everybody says I want to look at the definition of a pipeline and have you deterministically tell me its lineage and I can't do that.
00:19:40.623 --> 00:19:50.971
Right, because there's a million things that a pipeline can be dynamic and so you say, well, there are N lineages related to this pipeline.
00:19:51.333 --> 00:20:00.691
Providing insight into that, I do think, is a thing that we can share back to the industry, and I'm excited to see that go forward, because integration is often a black box.
00:20:00.691 --> 00:20:03.836
Right, I'm the system on one side.
00:20:03.836 --> 00:20:09.635
I put data in and it didn't get to the other end the way I thought it was going to.
00:20:09.635 --> 00:20:12.683
Somebody on the other end said it didn't work right the way I thought it was going to.
00:20:12.683 --> 00:20:14.710
Somebody on the other end said it didn't work right.
00:20:14.710 --> 00:20:21.858
And now we have this like Spider-Man pointing at each other, where the ERP people point at the pipeline people and the CRM people point at the pipeline people.
00:20:21.858 --> 00:20:33.317
The pipeline people say, wow, I don't know right, like seeing into that and really being able to deterministically say for a single execution, I can tell you exactly what went where.
00:20:33.317 --> 00:20:43.195
I think that's a level you know, that's observability, that we get in some high-end software that we're going to see brought into our product and I think will be a huge change.
00:20:43.829 --> 00:20:53.717
Yeah, because you do get that moment of I don't actually know what happened when it ran, or I can't determine what will happen when it runs at some points in the future, because of these dynamic inputs.
00:20:54.097 --> 00:21:00.193
Well, and this is one of these, like the answer is both true and false simultaneously.
00:21:00.193 --> 00:21:04.910
Right, because technically you can get the pipeline execution.
00:21:04.910 --> 00:21:06.715
It's a great big JSON file.
00:21:06.715 --> 00:21:10.469
It's the same JSON file that I sort of start with, right?
00:21:10.469 --> 00:21:13.218
So here you can have it and you know.
00:21:13.218 --> 00:21:15.596
But can't you know how much can you derive from that?
00:21:15.596 --> 00:21:17.384
How easy is it to derive lineage?
00:21:17.384 --> 00:21:20.354
That's a completely different conversation, right?
00:21:20.354 --> 00:21:21.717
And how do you do that at scale?
00:21:21.717 --> 00:21:29.771
You know, I think moving that up the food chain is going to provide a lot of value, absolutely Quite different topic of question or conversation.
00:21:30.012 --> 00:21:31.896
I'm seeing again the larger accounts.
00:21:31.896 --> 00:21:40.236
The environmental discussion is not as simple as test dev prod pre-prod we also have.
00:21:40.236 --> 00:22:00.478
Because of the nature of our architecture we're not a single shared pool environment, so we end up having, in some places we'll have a shared environment for less complex We'll also have something like an ultra environment for obvious reasons, where it's got a very different physical dynamic.
00:22:00.478 --> 00:22:03.638
Some other accounts actually split, jolt out.
00:22:04.311 --> 00:22:22.894
I have worked with projects where we've also categorically said these other sub-platforms, so think CRM supply chain, will be considered to be mission critical and therefore they should have a separation of concern on resource management and therefore they are their own landscape.
00:22:22.894 --> 00:22:24.378
What's your thoughts on that?
00:22:24.378 --> 00:22:42.115
Because at the moment, going back to CICD and automation around this, we don't seem to have a point of view yet of tagging the lifecycle of the platform so that you could easily automate characteristics through the lifecycle of moving from one to an environment to another.
00:22:42.115 --> 00:22:59.382
There's a lot of human IP would be needed to carry through this pipeline is of a certain characteristic criticality, ownership and the endpoint will be platform A rather than the simple test dev pre-prod that we're all used to.
00:22:59.382 --> 00:23:02.470
What have you seen and what's your thoughts around that challenge?
00:23:03.010 --> 00:23:17.273
I mean, I think there's another dimension to it as well, which, as a middleware team, you know like some of this is consideration of the sort of the usage by the target customer, right?
00:23:17.273 --> 00:23:27.792
So if you're using SnapLogic and you're two citizen developers and you've got 40 workflows, that's a very different ecosystem from you.
00:23:27.792 --> 00:23:39.305
Know, I have 50 developers or 100 and I've got, you know, n environments and you know what you described and you know what's the jumping point for that, I mean in complexity.
00:23:39.305 --> 00:24:09.800
In a previous role I thought a fair bit about this, right, and I think you one huge inflection point is any decent sized operating company that has bought another one, right, as soon as you do, that, stuff starts to get fun in terms of environmental controls, because now you have a ton of systems that are on different life cycles and, and so you'd love to say dev, test, qa, prod, but the reality is and this was a big surprise to me, right?
00:24:09.820 --> 00:24:26.169
You know, stood up a middleware team and we had four or five environments and then what we figured out was that a lot of our constituent participating systems could not keep up with that level of parity and in many cases it's not worth it, right?
00:24:26.169 --> 00:24:39.645
So, like it's a small labeling system and they charge you per instance and they don't care if it's DevTest, qa or Prod, it's an equal charge, right, and there's two users and it's really lightweight and you go well.
00:24:39.645 --> 00:24:48.632
On the other side, by the way, is ERP, and there are eight instances and those folks can snap their fingers and make a new one.
00:24:48.632 --> 00:25:00.287
Like, people think about ERP as this big burly thing, but a lot of ERPs are insanely good at like being able to fraction off a new instance with data and everything and then start working.
00:25:00.988 --> 00:25:02.752
Right, as you know, I was at SAP.
00:25:02.752 --> 00:25:06.488
I encountered one of their largest accounts.
00:25:06.488 --> 00:25:14.444
Just to give you how big big can get, they were running 128 ERP landscapes.
00:25:14.444 --> 00:25:16.406
Yeah, so not instancesP landscapes.
00:25:16.406 --> 00:25:17.907
Yeah, so, not instances landscapes.
00:25:17.907 --> 00:25:21.832
Now, the reason was they were running from the free live environment.
00:25:21.832 --> 00:25:24.074
They had a five-layer strategy.
00:25:24.354 --> 00:25:53.583
So if you think about instance management, yeah, they were in excess of 600 non-production environments that this whole discussion would actually have to cope with even when you get to like way smaller than that, though, right like, you don't have to get very big before you realize that it's not just instant strategy, but it's how do you use your middleware as an equalizer, right like, how do you use your middleware as a traffic cop?
00:25:53.583 --> 00:26:18.016
I mean, it got a little weird, but we did a big staged cutover where we used middleware, we used integration as the strategic weapon in large acquisition integrations, you know, bringing new ERP instances in and new data feeds in and new, you know, api endpoints online that are federated across.
00:26:18.016 --> 00:26:21.758
Whatever's your middleware is your secret weapon there.
00:26:21.758 --> 00:26:34.532
But it means you think about, you know we would say, hey, listen, at noon we're going to switch over and it's no longer connected to this instance, it's connected to that one.
00:26:34.532 --> 00:26:39.433
So everybody in the ecosystem knows we've switched and we would play traffic back and forth and we use.
00:26:40.214 --> 00:26:54.045
We had a piece of software with a database in it, with a decent little user interface that we use to manage that and, like, all participants could get in and understand what was connected to where, when, and it was huge.
00:26:54.045 --> 00:27:02.032
It was a huge sort of enabler, because it's not everybody's going to be able to meet the multi-environment requirements.
00:27:02.032 --> 00:27:09.422
But you can use your middleware, right like you can use snap logic to have n and then make those like for a given environment.
00:27:09.422 --> 00:27:11.346
Then you go horizontal right.
00:27:11.346 --> 00:27:15.001
So in a given environment, which one of the nodes am I supposed to be going to?
00:27:15.001 --> 00:27:17.506
And that that's your whole ultra versus.
00:27:17.506 --> 00:27:27.771
You know whatever conversation short of humans, right Like I think you have to set that up per environment or per customer, like that's so customized.
00:27:27.771 --> 00:27:30.145
I don't know a way to generalize that right.
00:27:30.346 --> 00:27:31.509
Yeah, it's not going to be portable.
00:27:31.509 --> 00:27:43.511
It's too specific, although some of the stuff that's happening now can at least help you with doing things like mocking up your test data so that you can maybe postpone that moment of reckoning a little bit into the future.
00:27:43.511 --> 00:27:56.473
It's one of the things that I like about SnapGPT that you can make me a dozen records that look like the output of System X, so that you can play around and get things right before you even start connecting to the dev environment, and so on and so forth.
00:27:57.121 --> 00:28:03.233
So we've been talking about how standard engineering practices apply to low-code no-code.
00:28:03.233 --> 00:28:14.211
If you were to approach a platform like SnapLogic, where you've come from a very code-orientated point of view, do you think there are things that you shouldn't be trying to apply to low-code no-code?
00:28:14.211 --> 00:28:16.724
You could end up risking undermining the value.
00:28:16.724 --> 00:28:26.573
So bit of a philosophical question, but I'm always a great believer in the sessions of okay and what shouldn't be done as much as what should be done.
00:28:26.819 --> 00:28:47.967
You know I won't name any names, but you know I listened to a recording from a customer prospect that had done a pretty intensive kind of evaluation of SnapLogic for their demands and said you know, we're not, we don't want to do it, we're not, we don't want to use it, and we have very detailed feedback.
00:28:47.967 --> 00:28:48.367
Why?
00:28:48.367 --> 00:28:51.698
And I was very interested in it?
00:28:51.698 --> 00:28:53.462
Right, because I had my own opinions too.
00:28:53.462 --> 00:28:56.587
And I was very interested in it.
00:28:56.587 --> 00:28:58.250
Right, because I had my own opinions too.
00:28:58.250 --> 00:29:02.315
And I think the short answer comes to contemplating the right levels of abstraction.
00:29:02.315 --> 00:29:06.101
And what do I mean by that?
00:29:06.101 --> 00:29:21.670
Right, if you want to use SnapLogic to connect your ERP to your CRM and your PLM and file exports and to your EDI providers, right, and there's infinite in and out connections.
00:29:21.670 --> 00:29:22.820
You know.
00:29:22.820 --> 00:29:31.319
You want to use SnapLogic to build a data warehouse and move data over and then orchestrate SQL against it All those great use cases.
00:29:31.319 --> 00:29:53.382
If you want to make pipelines that have for loops with complex, true false conditions and you want to do things that look like try catch exception behavior with complicated retries that you're programmatically controlling, you know there's this thing where and it's I don't know.
00:29:53.382 --> 00:30:09.630
It's a little Zen to say right, but like, if you let software engineers try to use SnapLogic like software, like try to make pipelines that look like programs, they will not have a good experience and the pipelines will be unpleasant to look at and debug and use.
00:30:09.710 --> 00:30:10.583
Right, like you have to.
00:30:10.583 --> 00:30:16.586
You have to understand and get your head into the level of abstraction that it presents.
00:30:16.586 --> 00:30:22.390
And if you've got some complexity or some weirdness to hide sometimes it's okay.
00:30:22.390 --> 00:30:31.073
Well, go use the APIM product, right, because you want to orchestrate APIs, but some of those have these more complex workflows behind them.
00:30:31.073 --> 00:30:33.528
Okay, well, let's go stand up some APIs.
00:30:33.528 --> 00:30:36.067
Let's compartmentalize this thing.
00:30:36.067 --> 00:30:42.801
Let's make pipelines that are the right level of granularity, right, we're thinking about the right level of abstraction.
00:30:42.801 --> 00:30:52.049
So if you try to do programming things in a visual programming tool, it'll hurt, right, like, don't do that and that's blurry.
00:30:52.049 --> 00:30:52.309
I don't.
00:30:52.309 --> 00:30:58.848
I, that's blurry, but I'm interested if that resonates, right, like dominic's sort of smiling it does.
00:30:59.109 --> 00:31:24.391
So I'll give you one of my prior experiences, but also, uh, we've had one of our previous podcast guests was chris ward from amir's um services team, where we had an account where a one strategic division had chosen us another team that was very traditionally developer orientated and they could not get their heads around the mindset.
00:31:25.077 --> 00:31:30.089
And we put Chris Ward in because, like yourself, he has a massive.
00:31:30.089 --> 00:31:47.051
His prior history was very classic development around integration and he spent a lot of time almost coaching what you said like the Zen of it how to think about this thing, to get the best out of it, rather than having an argument about parity of mindset.
00:31:47.051 --> 00:31:48.685
That is exactly what he's saying.
00:31:48.685 --> 00:31:55.227
He's like I want it this way because this is how these my, my environment and tools and I think about development, and he sort of had to go.
00:31:55.227 --> 00:32:10.662
You need to go three levels higher because this will do it for you and then you will think about it differently when you want to apply your outcomes, but your outcomes will not be the thing that you've spent 10 years developing there's a pattern-based discussion that you get to have with.
00:32:10.903 --> 00:32:16.962
You know, in any decent ecosystem there's a pattern-based conversation to have with the other party as well.
00:32:16.962 --> 00:32:20.131
Right, and for us this was really interesting.
00:32:20.131 --> 00:32:26.771
You know, one of our biggest wins was being able to go to the various ERP teams, because we had several.
00:32:26.771 --> 00:32:36.323
Right, because you ought to and I mean different technologies too, not just different instances, but you ought to have all of the vendors and multiple instances of all of the vendors.
00:32:36.323 --> 00:32:40.761
We were able to go to those teams and say, hey, listen, you know, we have this middleware.
00:32:41.223 --> 00:32:47.625
We'd like to develop the two or three standard patterns for how we move data in and out of systems that look like.
00:32:47.625 --> 00:32:50.152
So now the COEs can interact with each other.
00:32:50.152 --> 00:33:02.530
Right, and there's this magnifying effect of oh, if I have any erp in the fleet and any crm in the fleet, I actually have patterns and recipes all the way through.
00:33:02.530 --> 00:33:10.400
And to dominic's point earlier, the big interesting thing about integration is that, you know, some of them do stay still for years.
00:33:10.400 --> 00:33:34.625
But, like, the integration platform is a very fast moving like, ideally it's a strategic enabler because you can go so fast at making, you know, making changes and rolling them through um, because ultimately the people on both ends are like you're the slow point in many cases right, like if you're not, then you can sort of move both ends along faster.
00:33:36.347 --> 00:33:41.496
And, just to round out, you talked about again the complexity in the UI.
00:33:41.496 --> 00:33:57.210
In a prior middleware company we had a famous customer that used a Flow UI and for 95% of its use cases it was brilliant.
00:33:57.210 --> 00:34:10.260
And then they complained about the last five and when we went in and reviewed it to do a translation, they were building integration pipelines that weren't tens or even a hundred.
00:34:10.260 --> 00:34:20.530
They would have been the equivalent of thousands of snaps and at that point the equivalent of yourself.
00:34:20.530 --> 00:34:32.813
Sir, basically said get down to the code split and analyze it at the code level, because no human being can look at this thing and actually understand what you're doing.
00:34:32.813 --> 00:34:35.409
And basically there was a whole interesting.
00:34:35.409 --> 00:34:41.407
They'd effectively built a gigantic application under the concept of it being an integration.
00:34:42.161 --> 00:34:49.409
Was this all a cunning plan to get approval to buy gigantic HD monitors for themselves?
00:34:50.141 --> 00:35:04.659
It might not have been, but again, it was just interesting about that mindset of trying to get to the yes, you could, but should you be doing it with a particular type of, as you say, abstraction and tooling?
00:35:04.659 --> 00:35:08.748
Um, when, when do you start almost getting diminishing returns?
00:35:08.748 --> 00:35:09.650
And because?
00:35:09.650 --> 00:35:23.664
Because the diminishing returns from the vendor I was at was, yeah, there were no meta-error statements that you could say somewhere in this 120 steps, something has gone wrong somewhere in here.
00:35:23.664 --> 00:35:34.527
Drill down to here, of course, what you got was one very little thing went red, but the ability to roll that back to understand really what was going on wasn't correct.
00:35:34.527 --> 00:35:37.650
But anyway, that's a side piece, indeed.
00:35:38.900 --> 00:35:40.909
No, this has been a great conversation.
00:35:40.909 --> 00:35:44.530
Do you have any closing thoughts, Brian?
00:35:44.530 --> 00:35:50.606
Things that you wish more people knew and thought about and considered in their use of SnapLogic?
00:35:50.606 --> 00:35:52.445
This one weird trick.
00:35:53.239 --> 00:35:54.302
This one weird trick.
00:35:54.302 --> 00:35:56.871
No, use source control.
00:35:56.871 --> 00:36:01.632
You know it sounds funny, but like somebody needs to.
00:36:01.632 --> 00:36:03.987
A thousand people need to hear that Use source control.
00:36:03.987 --> 00:36:08.128
If you don't know how to use source control, ask a developer.
00:36:08.128 --> 00:36:12.561
They'll probably be mean about it, but learn to use source control.
00:36:12.561 --> 00:36:13.585
That's life-changing.
00:36:14.581 --> 00:36:18.704
Indeed, and something will happen at some point and you'll be glad you did.
00:36:20.206 --> 00:36:27.681
I swear, if you can reduce most of what you do to text files that you can store in source control, it's solved an inordinate number of problems for me over the years.
00:36:28.684 --> 00:36:30.489
This is very true, Excellent.
00:36:30.489 --> 00:36:34.222
So this has been a fantastic and wide-ranging conversation.
00:36:34.222 --> 00:36:40.402
Thank you so much for taking the time out of your busy schedule to talk to us For the audience.
00:36:40.402 --> 00:36:46.365
Remember that this is only part of Enterprise Alchemist, what you're listening to right now.
00:36:46.365 --> 00:36:49.710
If someone sent you this episode, you can subscribe to the whole feed.
00:36:49.710 --> 00:36:58.744
Wherever good podcasts are downloaded and there is also a whole text-oriented side of things that goes on on the Integration Nation forum.
00:36:58.744 --> 00:37:09.380
A full transcript gets posted there if you prefer to read rather than listen, or if you want to go back to anything that we discussed, as well as links to anything relevant.
00:37:09.380 --> 00:37:19.710
I don't think we talked about any external resources in this episode, but when we do, we always link to them from there, as well as from the show notes right in your favorite podcast app.
00:37:19.710 --> 00:37:25.666
So thank you once again for listening and we hope you subscribe and check out upcoming episodes.
00:37:25.666 --> 00:37:27.188
Thanks all.
00:37:27.188 --> 00:37:27.869
See you later.