Tim:
0:01
Hi, and welcome to the podcast. We've got a really special guest, someone I know really well. We worked together, Cloudera. We had a lot of fun on some really cool streaming stuff. It's John Kuchmek, he's a principal solutions engineer at Cloudera, and we've done meetups together. So you've probably seen us around usually covering things in streaming, like Apache NiFi, Apache Spark, Apache Flink, Apache Kafka, other things. We did a meetup little while ago where we covered Pulsar and NiFi together for NFTs and solving the, our NFTs fake and is it a scan thing? And, you know, with the help of Kudu, maybe we did. So I think we'll have a cool discussion today and we're gonna cover some things we haven't covered in previous ones, basically the very powerful streaming ecosystem that Pulsar fits into the middle of. And we've got official connectors with Cloudera. So when we talk about this, these are things that have been tested by people out there. It's not just, you know, someone threw up a connector somewhere. It's been tested by, you know, serious people. So it's not just, uh, my friend David wrote something and put it out there and it works. It's out there for real. So the longest intro ever, and I barely talked about his background.
Matt:
1:30
Well let John introduce himself. You know, like John, John. Tell us about yourself.
John:
1:35
Sure. Uh, so yeah, I mean I've known Tim for, God, going on like seven, eight years, something like that. Um, it's been a long time. So I actually used to be a customer, so Tim was my solutions engineer. Uh, when, uh, I worked at a company called American Water, and so there I kind of really delved into a lot of the streaming ecosystem, being able to pull data out of operational technology, so SCADA systems, real time data feeds off of PLCs, things like that. We also built, uh, a lot of cool mechanisms to do, you know, incremental batch, not true change data capture, but out of SAP HANA before anyone had CDC for SAP on really. Uh, so, I've kind of, been able to get my feet wet in a lot of different streaming technologies. And now I'm on the other side of the fence where, where I get to help customers with their streaming problems. And so Tim and I have, I don't know, God, we've done how many, I mean, you know, probably 30 or so meetups together, I'd say over time, probably more than that.
Tim:
2:35
Felt like that.
John:
2:35
Uh, Any chance? Yeah. We're we only live like 45 minutes from each other too, so it's like, we're not that far. So, but yeah, so it's a little bit of a, my background, but before that I was, uh, I was actually a manufacturing engineer, so I worked at, uh, helicopter plant. Um, I, before that I was a research physicist, so, uh, I've got a, I've got a, like, a pretty diverse background, um, which I think is beneficial for being able to figure out, uh, how to help businesses implement, you know, streaming technologies and use cases.
Matt:
3:05
So when you talk about that, that American Water you said, right?
John:
3:08
Yeah.
Matt:
3:09
So was that like, you know, were you monitoring like water supply or, you know, flow through systems or?
John:
3:17
So the biggest thing that we worked on was we were actually pulling the sensor data off of the raw water intake facility and we, it's actually, there's an article about this. It's pretty cool. We, we kind of built a test lab to, to start to dope the water, like off to the side. Uh, obviously not going to the, to the actual water treatment facility, but we'd pull some water off and we'd set up the same system with the sensors and dope the water and actually try to figure out like certain signatures based on the sensors being matched. Could we match the chemical to figure out what was coming in? Because sensors aren't looking for specific chemicals. They're looking at turbidity. So like, you know how much, you know, there's, uh, part particulates within the water that's looking at, um, you know, uh, how much oxidation there is, right? So like there, there's temperature sensors. So all of these different things. So as you look the map, uh, you know, the different sensors together, we could try to figure out. Like, could we detect something before it even reached the water treatment center? And if it was detrimental, could we then stop it? Right? So that way we're not necessarily poisoning the well. So we did that. We did, we did a POC where we put Apache NiFi on, um, on iPads. So, uh, you know, you essentially could have the field, the field workers, we'd be able to, to track like, you know, all right, we'll match that with traffic patterns to figure out like if something's going wrong, could we reroute them or if there's bad weather coming. Also to monitor for safety. So there was a lot of things like we were very safety-driven and very, um, you know, health-driven with the water. So there was a lot of cool use cases that we utilized stream.
Matt:
4:58
Now. I, I've, Okay, so I know we, we've talked about this a few times, and Tim, you brought this up in the intro. So, you know, there, there are a lot of different Apache tool sets that are part of this tool chain. What is NiFi?
John:
5:12
So Apache NiFi was a tool that was out of the NSA, so I think it went open sourced in 2012, if I remember correctly. It actually stands for Niagara Files, so that's why it's, it's capital N, lowercase i, capital F, lowercase i, um, The NiFi is a UE-based data flow tool, so it makes it very easy. It's either low code, no code, depending on, you know, how you build your process to build a data flow, and it, it has a, a bunch of built-in connectors. I think there's like 328 or 330 roughly, you know, built-in connectors that they call processors. So each processor kind of behaves like a function, like from a code perspective, you can think of it as like, it, it does like something, as a function would do within code and then stitching all of the functions together. Your workflow kind of becomes like a, a, you know, a jar or a series of classes and then that goes into processor groups and so on and so forth. So, um, it makes it anyone with an engineering mindset, and I, and I tell this story. Was it two years ago now? Yeah, so, or maybe it was last year. I forget what it was. So I think it was two years, right when the pandemic started their international day of the girl I, my, my daughter is, at the time she was 13, so it'll be, it was a year ago. She was 13. We actually had her. Like we installed, uh, you know, NiFi on a Raspberry pie and we were pulling data out of a temperature sensor and we were doing some changes to it, like converting it. Everything would come in Celsius, so we would then convert it to Fahrenheit. But she was able to do all that in NiFi. And like she, although she does have the ability to code a little bit, I wouldn't say she's like an avid coder, um, but it, she does have a good engineering mindset. So I even say like, you know, it's possible for a 13 year-old to build a flow. Um, and so that way you really are kind of giving anyone with a capability that has an engineering mindset to be able to build a pipeline and start to get data in, you know, into, say, a database or whatever and be able to do the inspection that they need.
Matt:
7:10
Okay. Yeah, and I know like you and Tim both have like, worked on, you know, uh, the combination of multiple tools, Tim, you know, like being the Pulsar expert here, you know, like how does Pulsar and NiFi work together?
Tim:
7:26
Well, it, it's kind of the dream team. I mean, I, I like that combination so much. I mean, I have like a thousand pounds of stickers here for what I call the flipping stack. Cuz to me, Pulsar does a good part in the middle. But NiFi is that too, like how do I get stuff off of like these crazy little glasses? How do I get it off of water pipe? How do I get it out of some old systems, some logs, big data, like there's so many sinks, and it's so rich. It's so easy to do this without having to write a thousand lines of code. NiFi is my ingest point and often a sink, you know, Pulsar in the middle. I put some functions in there to enrich and transform, and then Flink after that or somewhere on the pipe. It's nice with Pulsar if I want 20 people to concurrently run it and do their own processing. I can. And then Flink and Flink SQL do, you know, joining streams, joining the crossing the streams, you know, getting all that data together. I mean, that's awesome. I mean, that was my talk at ApacheCon, was, uh, citizen streaming engineer, you know, how do you, uh, make it easy for people to write what used to be, you know, writing a real time streaming app, is something you're paying, you know, high end consultants of big money. But if I give you NiFi, Pulsar and Flink SQL, all you gotta know is drag and drop a little yaml, maybe to configure Pulsar. Maybe if you use, uh, the cloud, you just press a couple buttons. Uh, NiFi can create those topics on the fly the first time it publishes data in. And then on the other side you write some SQL to join things. You've built a real time scalable streaming app with like I said, as long as you have an engineering mindset, you don't need the super coding. That's why I said citizen streaming engineer, like they say, citizen data scientist. I don't need, you know, 10 years of programming, Java Scholar background. I think at, uh, when we have time at CubeCon, maybe I'll show Matt how to build and flip an app. Maybe we'll do that real time. I mean, it's, if you've done SQL and, and you can use, uh, Facebook, you could probably build a app. It's not, uh, the only part that gets tricky is sometimes people give you the worst data they could find. Like, I got some real time transit data and it's, it's some nasty RSS. And there's things I really need to pull apart. So there's some extra steps you could do in some code or in NiFi, but the steps, maybe, you know, you might have to look at the manual, go, Oh, I could do that or that, or look at an article that John or me or a Pierre's people have put out there and go, Okay, let me bring that in. NiFi is very modular, so I could easily pull in a processor group that does something. Very easy to patch together an app, Lego style.
Matt:
10:38
Okay, so if I can paraphrase what both of you have said is, NiFi is almost the, the user interface that you know, gives you the nice gooey. Pulsar is the kind of the highway to move stuff around behind the scenes. And then Flink is your reporting engine, lack of a better word.
John:
10:58
Yeah, I mean, I, I would say, yeah, for the most part, right? When you think of the pub-sub with Pulsar, it's not just, you know, moving it around, but it also gives like the capability to store the data for a temporary period of time, right? So NiFi is not meant to maintain the data, although it does persist at the disc to make it resilient. It's not meant to hold onto the data as soon as the flow completes. That's it. It's gone, right? The data does not meant to stay there. Whereas Pulsar, you might have people when they're consuming applications, be able to be consuming 'em at the rate that they want. So you might have someone say, I need it now, you might have someone say, I need it once a day. And so you can maintain that data in a pub-sub model to be able to, to pull it at the rate that you want. And Flink SQL, I mean, it's not just reporting and analytics, but it does some really cool stuff with, um, you know, essentially you could do a lot of, uh, of windows. So, for the very important for where I came from, being able to window data sets because, you know, it's one thing to understand what's happening at the point of time, but it's another thing to under understand what's happening, aggregates over a time window. So every minute, every five minutes, every 10 minutes, do I see the aggregations are going up or remaining consistent because you'll have, like sensor read fluctuations could actually go back and forth, and you'll kind of have like a maximum minimum. But the aggregation, the total aggregation should never reach, pass a certain threshold over a certain amount of time. So, you know, just another thing. So a little bit more than just, you know, the sequel aspect, but you know, Flink is a pretty robust, um, you know, tool to be able to, to read that data in real time.
Matt:
12:31
And Tim has decided to bring his cats on screen.
Tim:
12:34
They have decided to, uh, enter the facility. I think they thought something we were saying was stinking. I don't know, . Now, this, this guy's a new guy. He, he explores, What are you gonna do? So, so John, you know, we, you know, you mentioned some of the apps here are really IOT-sensor based. But, what about like more kind of modern web apps? How have you seen some of maybe like, you know, the stack used in, you know, more web type or end user type applications?
John:
13:07
So, I mean, you can build a lot of end user applications. There is actually, we have a flavor of Flink SQL. That we call SQL Stream Builder, that allows you to go ahead and create a materialized view of the data set, and so you can feed an application in real time that data feeds, so you could actually have like a point of time of, what things look like. Dashboarding is definitely an important thing. Now, pulling data off of, you know, web apps is definitely possible as well. Like you can, if there's rich APIs to be able to pull data from, like NiFi is not just about sensors to pull data. NiFi, like I said, has a rich set of connectors. So you can have both a push or a pull model depending on what you want. So you can actually have, uh, NiFi be a listener. So if you want it to do something within an application and push the data to say, a website, you can use NiFi as that endpoint. It go ahead and grabs the data set and then you could do some transformation on it before you push it up into Pulsar, uh, and then potentially, you know, read it from Pulsar for Flink and then create that materialized view to dashboard exactly the thing that you've been pushing.
Matt:
14:13
Okay. The cat has made an appearance again. What's the cat's name? Tim?
Tim:
14:17
This one, his unofficial name is Splu. His real name is Ash.
Matt:
14:23
Ash. Okay. Okay.
Tim:
14:25
Yeah, he's very friendly. You're very friendly, right? Yeah. Yeah. Sweet. I think he could write one of these apps. He was like,
Matt:
14:31
Oh boy. So I guess that's Tim's challenge to, to get that app under. If I could get a cat to write an app. Oh, that's not a bad, uh, talk title for a conference.
John:
14:41
Good. That would be good.
Tim:
14:41
Yeah. Tim could teach a cat to build an app with flippers there. We're,
Matt:
14:46
Well, you just need to get the sensors around the cat's neck and the collar, and then have, you know, it wander around so you can track your cat while you're at a conference.
John:
14:54
Someone did, someone did that with a video? Yeah. Yeah. Someone did that with a video feed. I think it was a dog where they actually followed the bell around on video. But we could potentially bring that in real time. And then do you, you do one of your, you know, your TensorFlow processor. In NiFi where you actually, the recognizing exactly what the image is, you know, as it comes in essentially doing character recognition or doing, you know, um, you know, object recognition. Be pretty real time app.
Tim:
15:21
Yeah, not a bad idea. I could hook that up to the, uh, I got Nvidia, uh, uh, yeah, pretty powerful guy over there. That would run that real time maybe with PyTorch or so. Hmm.
Matt:
15:36
So we, you know, as we're talking about, like, you know, monitoring our cats, I don't have a cat, my daughter does. So I guess I could monitor my daughter's cat, but I don't really care. Um, so , what other weird use cases have you seen Pulsar and NiFi and, and the whole stack be used for? Like, are there some other eclectic use cases that you're like, Yeah, this is a really weird one, but it, it kind of makes sense.
John:
16:03
I'll let you go first. Tim. I'm trying to think of. So I don't think anything's weird. Like I've,
Matt:
16:09
You can say cool. You can say cool.
John:
16:11
That's fair enough. Go ahead and give me a cool one first, Tim.
Matt:
16:15
Well, when we were running the meetups, I used NiFi to, to find and download live music streams. That's kind of, kind of weird. I mean, cuz NiFi could be a general connection of anything. I mean, the NFT thing we do is a little weird, weath- doing real-time weather, pulling all the weather in from the country parts. Okay. Okay. Okay. Wait, wait, wait, wait, wait. You know, so when you say the NFT thing we did do is kind of weird, what is the NFT thing you do? Like, you can't just like, like, Oh, it's just this weird thing. Like you have to explain what the weirdness is.
John:
16:55
So we, Tim, Tim and I did this meetup. It's actually a, you can go to YouTube, I'm pretty sure. So I, I don't know if this podcast has the capability to give links somewhere, but I'm pretty sure we can get you that link. Um, we went in just to look at NFTs. Now I kind of come from a background where I think NFTs are kind of BS a little bit, Um, you know, , uh, so I'm biased. Uh, but you know, we, Tim was actually able to pull in, um, NFT data into Pulsar. I store it, and then NFI was actually taking that data in real time and streaming it. Now I did some transformation to the data set because it came in via an API. So it was JSON. I flattened the JSON out, uh, to give me a nice row column type format. Uh, but it also included, uh, the links to the gift images. And so we have internally a tool called, uh, data visualization, which is really nice. It's a great dashboarding app. And so I ended up creating a table so it would stream a Kudu, Apache Kudu, so as other open source tool, uh, with Apache Powa on top of it, which, you know, Kudo is the storage layer and Powa's the sequel layer. And so now I have a good table and I use the data visualization application to show, uh, all of the, the different NFTs and it actually showed the gif image of the NFT Plus who, who bought the NFT, who sold the NFT, what the price was for, and the thing that we found out was that the same person was buying and selling the NFT. Um, it was the same user, which means that they're artificially driving up demand. At least that's what it looked like. I said. Obviously we need to do additional inspection, but it doesn't seem like there's a ton of regulations around the buying and selling of NFTs like there is for stock. So that's something that they call pump and dump in the stock world, they artificially drive up demand for the stock and then they dump it all, and then they make a ton of money. So it looks like that's what was happening with the NFTs, which is why I, you know, we, why in the beginning we said, uh, you know, NFTs might not be, you know, as big of a driver as people think.
Matt:
18:54
Ah, that's interesting. Um, you know, I think, I think you get a lot of that, you know, in, in the new kind of digital currency side as well, whether it's NFTs or even crypto.
John:
19:05
Yeah.
Matt:
19:05
There's, there's a lot of crypto out there that's, you know, questionable. Um,
John:
19:09
Well look at the, what is it? Kardashian had to pay, I forget how many millions, was it? 1 or 2 million dollars because of what she was posting on Twitter. I mean,
Matt:
19:18
It's, it's a weird space now, right? It is. Yeah. But, but those are, those are use cases that you two came up with from a customer perspective or out there in the real world, like actually running today. Are there anything that stands out as like a very unusual or interesting use case?
John:
19:37
Interesting. I mean, there's a lot of interesting use cases, so being able to marry, you know, weather data, traffic data and crime data was something that we worked on as well. Um, I thought that was kind of neat, you know,
Matt:
19:50
So like, do do more crimes get committed when the weather's bad and people are stuck in traffic?
John:
19:55
No. So, so more of like everything was around the safety of field service workers.
Matt:
20:00
Hmm.
John:
20:00
Um, and so as they, you know, as you go through like traffic, having traffic, you know, Google Maps will reroute you to where you want to go or whatever maps engine that you use. They, they recognize traffic patterns and they go, Hey, we found a better way, so we're gonna reroute you. So that's one thing. But taking that and then saying, Hey, there's a storm that's coming, you know, and being able to figure out like if the storm is localized to a specific area, um, obviously this might cause flooding in an area. You know, being able to figure that stuff out and then reroute them yourself was something that we thought about as well as a crime. So if there's actively something going on, like there's, you know, sometimes you'll find out about active shooting events where all of a sudden now there's police on scene and that's not actually captured in like a real time traffic app. So now you can say, Hey, stay away from there. It's not safe, you know, so reroute them a different way or if you're, you know, sending like a worker to an area, like for a house for servers, and all of a sudden there was a crime event that was happening nearby. Okay. Like, we don't wanna send you right now, you know, we'll, we'll basically reschedule that work order for another day.
Matt:
21:07
Hmm.
John:
21:08
Or if it was a high crime area, make sure that you match it to be able to send them early in the day rather than, you know, later in the day. Or, you know, being able to map time to, to probably, like there was a lot of cool stuff that I've seen used. Um, I've also seen it used for like pulling data information off an Apple watches to help with understanding health. I mean, I can't really name the client, but like, you know, they're. There's, there's, there's a tool out there called Lytic, by the way. Um, that does, uh, it, it helps pull the data off the Apple Watch itself, but it sends a, what's called an HTTP SSE, to a server sent event. So, you know, NiFi can be positioned to go ahead and grab that server, sent event stream and then be able to push it into something like Pulsar downstream, so that way you can have people consume it and build applications off of it later on. So I, I've seen a lot of cool stuff. I mean. I can't say all of it unfortunately, because some of it is, uh, very proprietary. But, you know, for the, for the cool stuff that I, I've seen or it's been talked about in the news or something, uh, you know, those, those are things that I can kind of, kind of discuss.
Matt:
22:12
Okay. Um, so, you know, it, it, it's interesting, um, I think that a lot of people that I have talked to, consider, you know, when we talk about like, you know, like, like this, this big data type problem as large company problems, but it kind of permeates all sizes, right? So smaller companies, mid-size companies, everybody can make use of these tools.
John:
22:34
Um, I agree. I mean, so I've dealt with, I've actually worked with some of the largest customers that we have, and I've also worked with some of the smallest customers that we've had. And it, it's not necessarily the size of the customer. I mean, everyone realizes that data is important now, and making decisions in real time is become even more important. Um, you know, the old days of ETL and batch workloads and having data. Like, be staled 24 hours, that, that's not gonna help. Now obviously larger corporations like financial sector to be able to do fraud detection in real time, that's a good use case that we see. Um, you can actually get the data in a real time feed and then try to do some intelligence on that data set and then go ahead and say, Hey, this looks like it's a potential fraud case. Uh, I don't know about you guys, but you know, we, we work with some banks and I have, you know, my bank has alerted me like, You know, multiple transactions from different places within the same amount of time. Is this, you call us right away, if not, we're gonna turn off your credit card. Right. And then if they temporarily shut down my credit card until I give'em a call, now that I think that's great because now I'm not having to worry about money being taken outta my account. Um, you know, I can give them a call back. But then really small companies that we've dealt with that, you know, they're doing, mom and pop type shop stuff to be able to understand like how they're machining data, you know, or how they're machining parts a little bit better, like, you know, there are smaller companies that produce like maybe just one part, and everyone in the world uses that one part kind of deal. They're not a huge company, right? They're, they're not cranking out millions of different products. They're specialized in one area, but they still need to understand, because if that machine goes down and they can't actually, you know, service it soon enough, then what ends up happening is, is that they lose, you know, ends up losing millions of dollars, which is their bread and butter. It doesn't matter anymore. I mean, everything is censored and with networks getting better and better, we're moving to 5G, I mean, IOT is gonna take over the world, my opinion.
Matt:
24:42
Yeah. Yeah. So as you do work with these, these folks, and Tim, you can jump in too if you want to answer this or you know, John, either one. What are the biggest mistakes that you see people make over and over again?
John:
24:55
You go first. Tim. I'm laughing because I've seen a bad, a lot of bad mistakes.
Tim:
25:02
I got one. Remember that, uh, truck Chassis company?
John:
25:06
Yeah.
Tim:
25:07
I mean, at one point, very small IT group. And Matt, we've talked about them before because of why they shut down their IT. But, uh, when they first started, the value they got out of, uh, the modern streaming stack is using NiFi to automate things they did by hand. You know, like reading an email, taking a file out, downloading it, like that part they did well, but then they took, a really antiquated system and tried to move it without improving it and changing it to a modern one. They're like, You know, these old mainframe queries on our hundred records. I wanna do that on this giant set of data, or this giant streaming set of live data are gonna bring in more streams of data. And I wanna do it the same old way. I wanna do those same old reports and print 'em out, and no one's gonna read 'em. If your mind's not there, the app's not gonna be.
John:
26:04
I'd say that's the biggest thing that we see is people coming from legacy systems moving to modern architectures are not rearchitecting. They want everything to be the exact same as it was before, and it can't be right. There's a reason why it's not. It's it, There's a reason why it's antiquated. Right? There's a reason why it's legacy. There's a reason why no one supports it anymore. Um, it's because everyone's gotten away from that type of architecture. Uh, beyond that, you know, doing things. So doing things like you've always done, you know, when everyone says that's how we've always done it, it's, it's my, my least favorite lines ever to hear anyone say in the world. Cuz I'm like, Well, that didn't work for you before, so why would you continue to do it the same way? And then the other thing is, is that especially with NiFi, so Mark Payne, who is, uh, one of the co-creators of NiFi, and obviously a PMC, he put out a series of videos called anti-patterns. So it's five videos where he actually goes and he, he figures out, or tells people like, here are the things that we see in the field that people should never do. So NiFi has evolved a ton over the years, when it first started, everything was about, you know, the flow file or the stream. Now they realize that, it needs to also have a lot of record capabilities. So you see like old-school methodologies or like, you know, if you go to Stack Overflow, it's like, well if you just do this, this and this, this will work. And you're like, Great. That was true like six years ago, but like now that's not the best way to do it. You know, there's a lot that's changed. So like, updating and enriching records. There's literally a processor called update record, whereas before, back in the day, you usually had to then pull it up into an attribute, use the expression language to go ahead and modify the data, then pull it back down. And what people didn't realize is like in order to do that with a record, you have to split the file. So if it's millions of records, you now have gone from one record to millions, they share the attributes across the line except for the ones that you wanna modify. All of that gets stored in a JVM. So, you know, you, you essentially get this memory overload that's happening. So there's just so much like bad patterns that occur when people build flows, and it's, it's primarily because they haven't really been up to speed on the net new stuff. So for me, like, you know, Watching Mark's videos. Like whenever I go to a customer and I see like bad things, I'm like, You need to watch this. Even when I'm just actually there, just getting introduced to NiFi, I'm like, Before you build anything, please watch these videos. Like they're very important videos to watch so you don't make mistakes.
Matt:
28:42
Hmm.
Tim:
28:43
That is critical.
Matt:
28:47
Yeah. Okay. Cool, cool. So not only are there mistakes that people make, they're, they just don't understand the technology sometimes. Yeah. And that's another area that I see quite often, right? So when, when you've got someone who is migrating from a legacy system, you know, and the re-architecture, I, I know it's a big thing. I've seen it myself with other, other technologies as well. Um, but I think it's also just the knowledge gap. So are there a few features or things that are underutilized or under known about like the hidden features or got, Everybody should know this, but they don't. Let me tell everybody what this thing is.
John:
29:25
So through the UI perspective, I, a lot of people don't realize this, but when you click on a processor, when you right click on a processor, there's a view usage category. That view usage category takes you directly to the documentation of that processor.
Matt:
29:40
Hmm.
John:
29:40
And I couldn't tell you how many times, like I, I've worked with a customer and they're like, Well, I don't understand what this means. Right. So now you highlight over it and it usually does give a description, but it literally having that capability to go right to the doc.
Matt:
29:52
Hmm.
John:
29:52
From that specific processor, not having to Google it. . Um, and especially like a guy like, you know, Tim or I have been using NiFi for a very long time, so our cache on our browser might take us back to like, you know, NiFi 1.1, We're on 1.17, about to be 1.18 out.
Tim:
30:09
Oh, 1.18 is out. Man, I'm running.
John:
30:10
Oh, 1.18 is out. There you go. See, I'm a little behind of the time, so I've been traveling, so I apologize. But yeah,
Tim:
30:15
Like two weeks. . John: Well, I've been on the road like the, you know, you look at, you look at that, you know, there's been so many changes that have occurred, like, and those processors change a little bit. Like they're not drastic changes, but there might be net new functionality. Another thing that people don't use a lot, um, is parameters. Um, we are trying to push, push, push, push, push. The utilization of parameters. Back in the day, and this goes back to the what used to be done versus now is variables were the thing. So everyone used variables, but the problem is, is variables, you know, don't have the same richness that a parameter does. So if you're using a variable before at NiFi, you need to use a parameter. If you have things that you wanna port over easy from like one environment to another, use parameters. Because if the parameters set, if you're using parameters, you can move that flow relatively easy, and then point to like maybe a file and system or something. And then essentially, you know, you don't have to change every processor where like you have a URL or you have a path. Right. If that changes on the new system.
Matt:
31:22
Oh, cool. Yeah, I mean, very useful. They, they seem like common sense things, but they're usability of, you know, features like that. They make a lot of difference in people's lives on a day-to-day basis.
John:
31:34
Yeah. Yeah, Completely agree.
Matt:
31:36
So, Tim, final thoughts from you.
Tim:
31:40
Well, I think the, it goes down to, you know, when, when you have someone who's got a lot of experience with it, listen to 'em. Uh, you know, if you have, if you have the pleasure of having someone like John Kuchmek in as, uh, you know, when you're a potential customer or customer or you're at a meetup or an event, you know, listen to what they're saying and, you know, ask for things like we were talking about today, anti-patterns. You know, where, where's the best place to get information. You know, what should I use together and what should I not like? People ask me, should I do everything in Pulsar? No. There're things that go in NiFi. There're things that go in Flink. There're things that go in Spark. If you try to, like, I've got the one thing that does everything well, you know, a Swiss Army knife works, but you, if you built a house with it, you're spending a lot of time and you're suffering. Don't, just because it can do it, you know, a brick, I could use a brick to, uh, tear down a house, but you know, there might be some, uh, I might get in trouble, you know?
John:
32:47
Yeah.
Matt:
32:49
Makes sense. So, John, thanks for hanging out with us today. We appreciate you sharing some knowledge with us and telling us a little bit about NiFi and some of the other experiences you've had in the background here.
John:
33:00
Yeah. Cool. Well, thanks for having me. It's fun. I really appreciate it.
Matt:
33:04
Alrighty. Well, everybody, until next time, don't forget to subscribe, like, and follow us. Uh, we would love that. So, we'll catch you later.
0:01
Hi, and welcome to the podcast. We've got a really special guest, someone I know really well. We worked together, Cloudera. We had a lot of fun on some really cool streaming stuff. It's John Kuchmek, he's a principal solutions engineer at Cloudera, and we've done meetups together. So you've probably seen us around usually covering things in streaming, like Apache NiFi, Apache Spark, Apache Flink, Apache Kafka, other things. We did a meetup little while ago where we covered Pulsar and NiFi together for NFTs and solving the, our NFTs fake and is it a scan thing? And, you know, with the help of Kudu, maybe we did. So I think we'll have a cool discussion today and we're gonna cover some things we haven't covered in previous ones, basically the very powerful streaming ecosystem that Pulsar fits into the middle of. And we've got official connectors with Cloudera. So when we talk about this, these are things that have been tested by people out there. It's not just, you know, someone threw up a connector somewhere. It's been tested by, you know, serious people. So it's not just, uh, my friend David wrote something and put it out there and it works. It's out there for real. So the longest intro ever, and I barely talked about his background.
Matt:
1:30
Well let John introduce himself. You know, like John, John. Tell us about yourself.
John:
1:35
Sure. Uh, so yeah, I mean I've known Tim for, God, going on like seven, eight years, something like that. Um, it's been a long time. So I actually used to be a customer, so Tim was my solutions engineer. Uh, when, uh, I worked at a company called American Water, and so there I kind of really delved into a lot of the streaming ecosystem, being able to pull data out of operational technology, so SCADA systems, real time data feeds off of PLCs, things like that. We also built, uh, a lot of cool mechanisms to do, you know, incremental batch, not true change data capture, but out of SAP HANA before anyone had CDC for SAP on really. Uh, so, I've kind of, been able to get my feet wet in a lot of different streaming technologies. And now I'm on the other side of the fence where, where I get to help customers with their streaming problems. And so Tim and I have, I don't know, God, we've done how many, I mean, you know, probably 30 or so meetups together, I'd say over time, probably more than that.
Tim:
2:35
Felt like that.
John:
2:35
Uh, Any chance? Yeah. We're we only live like 45 minutes from each other too, so it's like, we're not that far. So, but yeah, so it's a little bit of a, my background, but before that I was, uh, I was actually a manufacturing engineer, so I worked at, uh, helicopter plant. Um, I, before that I was a research physicist, so, uh, I've got a, I've got a, like, a pretty diverse background, um, which I think is beneficial for being able to figure out, uh, how to help businesses implement, you know, streaming technologies and use cases.
Matt:
3:05
So when you talk about that, that American Water you said, right?
John:
3:08
Yeah.
Matt:
3:09
So was that like, you know, were you monitoring like water supply or, you know, flow through systems or?
John:
3:17
So the biggest thing that we worked on was we were actually pulling the sensor data off of the raw water intake facility and we, it's actually, there's an article about this. It's pretty cool. We, we kind of built a test lab to, to start to dope the water, like off to the side. Uh, obviously not going to the, to the actual water treatment facility, but we'd pull some water off and we'd set up the same system with the sensors and dope the water and actually try to figure out like certain signatures based on the sensors being matched. Could we match the chemical to figure out what was coming in? Because sensors aren't looking for specific chemicals. They're looking at turbidity. So like, you know how much, you know, there's, uh, part particulates within the water that's looking at, um, you know, uh, how much oxidation there is, right? So like there, there's temperature sensors. So all of these different things. So as you look the map, uh, you know, the different sensors together, we could try to figure out. Like, could we detect something before it even reached the water treatment center? And if it was detrimental, could we then stop it? Right? So that way we're not necessarily poisoning the well. So we did that. We did, we did a POC where we put Apache NiFi on, um, on iPads. So, uh, you know, you essentially could have the field, the field workers, we'd be able to, to track like, you know, all right, we'll match that with traffic patterns to figure out like if something's going wrong, could we reroute them or if there's bad weather coming. Also to monitor for safety. So there was a lot of things like we were very safety-driven and very, um, you know, health-driven with the water. So there was a lot of cool use cases that we utilized stream.
Matt:
4:58
Now. I, I've, Okay, so I know we, we've talked about this a few times, and Tim, you brought this up in the intro. So, you know, there, there are a lot of different Apache tool sets that are part of this tool chain. What is NiFi?
John:
5:12
So Apache NiFi was a tool that was out of the NSA, so I think it went open sourced in 2012, if I remember correctly. It actually stands for Niagara Files, so that's why it's, it's capital N, lowercase i, capital F, lowercase i, um, The NiFi is a UE-based data flow tool, so it makes it very easy. It's either low code, no code, depending on, you know, how you build your process to build a data flow, and it, it has a, a bunch of built-in connectors. I think there's like 328 or 330 roughly, you know, built-in connectors that they call processors. So each processor kind of behaves like a function, like from a code perspective, you can think of it as like, it, it does like something, as a function would do within code and then stitching all of the functions together. Your workflow kind of becomes like a, a, you know, a jar or a series of classes and then that goes into processor groups and so on and so forth. So, um, it makes it anyone with an engineering mindset, and I, and I tell this story. Was it two years ago now? Yeah, so, or maybe it was last year. I forget what it was. So I think it was two years, right when the pandemic started their international day of the girl I, my, my daughter is, at the time she was 13, so it'll be, it was a year ago. She was 13. We actually had her. Like we installed, uh, you know, NiFi on a Raspberry pie and we were pulling data out of a temperature sensor and we were doing some changes to it, like converting it. Everything would come in Celsius, so we would then convert it to Fahrenheit. But she was able to do all that in NiFi. And like she, although she does have the ability to code a little bit, I wouldn't say she's like an avid coder, um, but it, she does have a good engineering mindset. So I even say like, you know, it's possible for a 13 year-old to build a flow. Um, and so that way you really are kind of giving anyone with a capability that has an engineering mindset to be able to build a pipeline and start to get data in, you know, into, say, a database or whatever and be able to do the inspection that they need.
Matt:
7:10
Okay. Yeah, and I know like you and Tim both have like, worked on, you know, uh, the combination of multiple tools, Tim, you know, like being the Pulsar expert here, you know, like how does Pulsar and NiFi work together?
Tim:
7:26
Well, it, it's kind of the dream team. I mean, I, I like that combination so much. I mean, I have like a thousand pounds of stickers here for what I call the flipping stack. Cuz to me, Pulsar does a good part in the middle. But NiFi is that too, like how do I get stuff off of like these crazy little glasses? How do I get it off of water pipe? How do I get it out of some old systems, some logs, big data, like there's so many sinks, and it's so rich. It's so easy to do this without having to write a thousand lines of code. NiFi is my ingest point and often a sink, you know, Pulsar in the middle. I put some functions in there to enrich and transform, and then Flink after that or somewhere on the pipe. It's nice with Pulsar if I want 20 people to concurrently run it and do their own processing. I can. And then Flink and Flink SQL do, you know, joining streams, joining the crossing the streams, you know, getting all that data together. I mean, that's awesome. I mean, that was my talk at ApacheCon, was, uh, citizen streaming engineer, you know, how do you, uh, make it easy for people to write what used to be, you know, writing a real time streaming app, is something you're paying, you know, high end consultants of big money. But if I give you NiFi, Pulsar and Flink SQL, all you gotta know is drag and drop a little yaml, maybe to configure Pulsar. Maybe if you use, uh, the cloud, you just press a couple buttons. Uh, NiFi can create those topics on the fly the first time it publishes data in. And then on the other side you write some SQL to join things. You've built a real time scalable streaming app with like I said, as long as you have an engineering mindset, you don't need the super coding. That's why I said citizen streaming engineer, like they say, citizen data scientist. I don't need, you know, 10 years of programming, Java Scholar background. I think at, uh, when we have time at CubeCon, maybe I'll show Matt how to build and flip an app. Maybe we'll do that real time. I mean, it's, if you've done SQL and, and you can use, uh, Facebook, you could probably build a app. It's not, uh, the only part that gets tricky is sometimes people give you the worst data they could find. Like, I got some real time transit data and it's, it's some nasty RSS. And there's things I really need to pull apart. So there's some extra steps you could do in some code or in NiFi, but the steps, maybe, you know, you might have to look at the manual, go, Oh, I could do that or that, or look at an article that John or me or a Pierre's people have put out there and go, Okay, let me bring that in. NiFi is very modular, so I could easily pull in a processor group that does something. Very easy to patch together an app, Lego style.
Matt:
10:38
Okay, so if I can paraphrase what both of you have said is, NiFi is almost the, the user interface that you know, gives you the nice gooey. Pulsar is the kind of the highway to move stuff around behind the scenes. And then Flink is your reporting engine, lack of a better word.
John:
10:58
Yeah, I mean, I, I would say, yeah, for the most part, right? When you think of the pub-sub with Pulsar, it's not just, you know, moving it around, but it also gives like the capability to store the data for a temporary period of time, right? So NiFi is not meant to maintain the data, although it does persist at the disc to make it resilient. It's not meant to hold onto the data as soon as the flow completes. That's it. It's gone, right? The data does not meant to stay there. Whereas Pulsar, you might have people when they're consuming applications, be able to be consuming 'em at the rate that they want. So you might have someone say, I need it now, you might have someone say, I need it once a day. And so you can maintain that data in a pub-sub model to be able to, to pull it at the rate that you want. And Flink SQL, I mean, it's not just reporting and analytics, but it does some really cool stuff with, um, you know, essentially you could do a lot of, uh, of windows. So, for the very important for where I came from, being able to window data sets because, you know, it's one thing to understand what's happening at the point of time, but it's another thing to under understand what's happening, aggregates over a time window. So every minute, every five minutes, every 10 minutes, do I see the aggregations are going up or remaining consistent because you'll have, like sensor read fluctuations could actually go back and forth, and you'll kind of have like a maximum minimum. But the aggregation, the total aggregation should never reach, pass a certain threshold over a certain amount of time. So, you know, just another thing. So a little bit more than just, you know, the sequel aspect, but you know, Flink is a pretty robust, um, you know, tool to be able to, to read that data in real time.
Matt:
12:31
And Tim has decided to bring his cats on screen.
Tim:
12:34
They have decided to, uh, enter the facility. I think they thought something we were saying was stinking. I don't know, . Now, this, this guy's a new guy. He, he explores, What are you gonna do? So, so John, you know, we, you know, you mentioned some of the apps here are really IOT-sensor based. But, what about like more kind of modern web apps? How have you seen some of maybe like, you know, the stack used in, you know, more web type or end user type applications?
John:
13:07
So, I mean, you can build a lot of end user applications. There is actually, we have a flavor of Flink SQL. That we call SQL Stream Builder, that allows you to go ahead and create a materialized view of the data set, and so you can feed an application in real time that data feeds, so you could actually have like a point of time of, what things look like. Dashboarding is definitely an important thing. Now, pulling data off of, you know, web apps is definitely possible as well. Like you can, if there's rich APIs to be able to pull data from, like NiFi is not just about sensors to pull data. NiFi, like I said, has a rich set of connectors. So you can have both a push or a pull model depending on what you want. So you can actually have, uh, NiFi be a listener. So if you want it to do something within an application and push the data to say, a website, you can use NiFi as that endpoint. It go ahead and grabs the data set and then you could do some transformation on it before you push it up into Pulsar, uh, and then potentially, you know, read it from Pulsar for Flink and then create that materialized view to dashboard exactly the thing that you've been pushing.
Matt:
14:13
Okay. The cat has made an appearance again. What's the cat's name? Tim?
Tim:
14:17
This one, his unofficial name is Splu. His real name is Ash.
Matt:
14:23
Ash. Okay. Okay.
Tim:
14:25
Yeah, he's very friendly. You're very friendly, right? Yeah. Yeah. Sweet. I think he could write one of these apps. He was like,
Matt:
14:31
Oh boy. So I guess that's Tim's challenge to, to get that app under. If I could get a cat to write an app. Oh, that's not a bad, uh, talk title for a conference.
John:
14:41
Good. That would be good.
Tim:
14:41
Yeah. Tim could teach a cat to build an app with flippers there. We're,
Matt:
14:46
Well, you just need to get the sensors around the cat's neck and the collar, and then have, you know, it wander around so you can track your cat while you're at a conference.
John:
14:54
Someone did, someone did that with a video? Yeah. Yeah. Someone did that with a video feed. I think it was a dog where they actually followed the bell around on video. But we could potentially bring that in real time. And then do you, you do one of your, you know, your TensorFlow processor. In NiFi where you actually, the recognizing exactly what the image is, you know, as it comes in essentially doing character recognition or doing, you know, um, you know, object recognition. Be pretty real time app.
Tim:
15:21
Yeah, not a bad idea. I could hook that up to the, uh, I got Nvidia, uh, uh, yeah, pretty powerful guy over there. That would run that real time maybe with PyTorch or so. Hmm.
Matt:
15:36
So we, you know, as we're talking about, like, you know, monitoring our cats, I don't have a cat, my daughter does. So I guess I could monitor my daughter's cat, but I don't really care. Um, so , what other weird use cases have you seen Pulsar and NiFi and, and the whole stack be used for? Like, are there some other eclectic use cases that you're like, Yeah, this is a really weird one, but it, it kind of makes sense.
John:
16:03
I'll let you go first. Tim. I'm trying to think of. So I don't think anything's weird. Like I've,
Matt:
16:09
You can say cool. You can say cool.
John:
16:11
That's fair enough. Go ahead and give me a cool one first, Tim.
Matt:
16:15
Well, when we were running the meetups, I used NiFi to, to find and download live music streams. That's kind of, kind of weird. I mean, cuz NiFi could be a general connection of anything. I mean, the NFT thing we do is a little weird, weath- doing real-time weather, pulling all the weather in from the country parts. Okay. Okay. Okay. Wait, wait, wait, wait, wait. You know, so when you say the NFT thing we did do is kind of weird, what is the NFT thing you do? Like, you can't just like, like, Oh, it's just this weird thing. Like you have to explain what the weirdness is.
John:
16:55
So we, Tim, Tim and I did this meetup. It's actually a, you can go to YouTube, I'm pretty sure. So I, I don't know if this podcast has the capability to give links somewhere, but I'm pretty sure we can get you that link. Um, we went in just to look at NFTs. Now I kind of come from a background where I think NFTs are kind of BS a little bit, Um, you know, , uh, so I'm biased. Uh, but you know, we, Tim was actually able to pull in, um, NFT data into Pulsar. I store it, and then NFI was actually taking that data in real time and streaming it. Now I did some transformation to the data set because it came in via an API. So it was JSON. I flattened the JSON out, uh, to give me a nice row column type format. Uh, but it also included, uh, the links to the gift images. And so we have internally a tool called, uh, data visualization, which is really nice. It's a great dashboarding app. And so I ended up creating a table so it would stream a Kudu, Apache Kudu, so as other open source tool, uh, with Apache Powa on top of it, which, you know, Kudo is the storage layer and Powa's the sequel layer. And so now I have a good table and I use the data visualization application to show, uh, all of the, the different NFTs and it actually showed the gif image of the NFT Plus who, who bought the NFT, who sold the NFT, what the price was for, and the thing that we found out was that the same person was buying and selling the NFT. Um, it was the same user, which means that they're artificially driving up demand. At least that's what it looked like. I said. Obviously we need to do additional inspection, but it doesn't seem like there's a ton of regulations around the buying and selling of NFTs like there is for stock. So that's something that they call pump and dump in the stock world, they artificially drive up demand for the stock and then they dump it all, and then they make a ton of money. So it looks like that's what was happening with the NFTs, which is why I, you know, we, why in the beginning we said, uh, you know, NFTs might not be, you know, as big of a driver as people think.
Matt:
18:54
Ah, that's interesting. Um, you know, I think, I think you get a lot of that, you know, in, in the new kind of digital currency side as well, whether it's NFTs or even crypto.
John:
19:05
Yeah.
Matt:
19:05
There's, there's a lot of crypto out there that's, you know, questionable. Um,
John:
19:09
Well look at the, what is it? Kardashian had to pay, I forget how many millions, was it? 1 or 2 million dollars because of what she was posting on Twitter. I mean,
Matt:
19:18
It's, it's a weird space now, right? It is. Yeah. But, but those are, those are use cases that you two came up with from a customer perspective or out there in the real world, like actually running today. Are there anything that stands out as like a very unusual or interesting use case?
John:
19:37
Interesting. I mean, there's a lot of interesting use cases, so being able to marry, you know, weather data, traffic data and crime data was something that we worked on as well. Um, I thought that was kind of neat, you know,
Matt:
19:50
So like, do do more crimes get committed when the weather's bad and people are stuck in traffic?
John:
19:55
No. So, so more of like everything was around the safety of field service workers.
Matt:
20:00
Hmm.
John:
20:00
Um, and so as they, you know, as you go through like traffic, having traffic, you know, Google Maps will reroute you to where you want to go or whatever maps engine that you use. They, they recognize traffic patterns and they go, Hey, we found a better way, so we're gonna reroute you. So that's one thing. But taking that and then saying, Hey, there's a storm that's coming, you know, and being able to figure out like if the storm is localized to a specific area, um, obviously this might cause flooding in an area. You know, being able to figure that stuff out and then reroute them yourself was something that we thought about as well as a crime. So if there's actively something going on, like there's, you know, sometimes you'll find out about active shooting events where all of a sudden now there's police on scene and that's not actually captured in like a real time traffic app. So now you can say, Hey, stay away from there. It's not safe, you know, so reroute them a different way or if you're, you know, sending like a worker to an area, like for a house for servers, and all of a sudden there was a crime event that was happening nearby. Okay. Like, we don't wanna send you right now, you know, we'll, we'll basically reschedule that work order for another day.
Matt:
21:07
Hmm.
John:
21:08
Or if it was a high crime area, make sure that you match it to be able to send them early in the day rather than, you know, later in the day. Or, you know, being able to map time to, to probably, like there was a lot of cool stuff that I've seen used. Um, I've also seen it used for like pulling data information off an Apple watches to help with understanding health. I mean, I can't really name the client, but like, you know, they're. There's, there's, there's a tool out there called Lytic, by the way. Um, that does, uh, it, it helps pull the data off the Apple Watch itself, but it sends a, what's called an HTTP SSE, to a server sent event. So, you know, NiFi can be positioned to go ahead and grab that server, sent event stream and then be able to push it into something like Pulsar downstream, so that way you can have people consume it and build applications off of it later on. So I, I've seen a lot of cool stuff. I mean. I can't say all of it unfortunately, because some of it is, uh, very proprietary. But, you know, for the, for the cool stuff that I, I've seen or it's been talked about in the news or something, uh, you know, those, those are things that I can kind of, kind of discuss.
Matt:
22:12
Okay. Um, so, you know, it, it, it's interesting, um, I think that a lot of people that I have talked to, consider, you know, when we talk about like, you know, like, like this, this big data type problem as large company problems, but it kind of permeates all sizes, right? So smaller companies, mid-size companies, everybody can make use of these tools.
John:
22:34
Um, I agree. I mean, so I've dealt with, I've actually worked with some of the largest customers that we have, and I've also worked with some of the smallest customers that we've had. And it, it's not necessarily the size of the customer. I mean, everyone realizes that data is important now, and making decisions in real time is become even more important. Um, you know, the old days of ETL and batch workloads and having data. Like, be staled 24 hours, that, that's not gonna help. Now obviously larger corporations like financial sector to be able to do fraud detection in real time, that's a good use case that we see. Um, you can actually get the data in a real time feed and then try to do some intelligence on that data set and then go ahead and say, Hey, this looks like it's a potential fraud case. Uh, I don't know about you guys, but you know, we, we work with some banks and I have, you know, my bank has alerted me like, You know, multiple transactions from different places within the same amount of time. Is this, you call us right away, if not, we're gonna turn off your credit card. Right. And then if they temporarily shut down my credit card until I give'em a call, now that I think that's great because now I'm not having to worry about money being taken outta my account. Um, you know, I can give them a call back. But then really small companies that we've dealt with that, you know, they're doing, mom and pop type shop stuff to be able to understand like how they're machining data, you know, or how they're machining parts a little bit better, like, you know, there are smaller companies that produce like maybe just one part, and everyone in the world uses that one part kind of deal. They're not a huge company, right? They're, they're not cranking out millions of different products. They're specialized in one area, but they still need to understand, because if that machine goes down and they can't actually, you know, service it soon enough, then what ends up happening is, is that they lose, you know, ends up losing millions of dollars, which is their bread and butter. It doesn't matter anymore. I mean, everything is censored and with networks getting better and better, we're moving to 5G, I mean, IOT is gonna take over the world, my opinion.
Matt:
24:42
Yeah. Yeah. So as you do work with these, these folks, and Tim, you can jump in too if you want to answer this or you know, John, either one. What are the biggest mistakes that you see people make over and over again?
John:
24:55
You go first. Tim. I'm laughing because I've seen a bad, a lot of bad mistakes.
Tim:
25:02
I got one. Remember that, uh, truck Chassis company?
John:
25:06
Yeah.
Tim:
25:07
I mean, at one point, very small IT group. And Matt, we've talked about them before because of why they shut down their IT. But, uh, when they first started, the value they got out of, uh, the modern streaming stack is using NiFi to automate things they did by hand. You know, like reading an email, taking a file out, downloading it, like that part they did well, but then they took, a really antiquated system and tried to move it without improving it and changing it to a modern one. They're like, You know, these old mainframe queries on our hundred records. I wanna do that on this giant set of data, or this giant streaming set of live data are gonna bring in more streams of data. And I wanna do it the same old way. I wanna do those same old reports and print 'em out, and no one's gonna read 'em. If your mind's not there, the app's not gonna be.
John:
26:04
I'd say that's the biggest thing that we see is people coming from legacy systems moving to modern architectures are not rearchitecting. They want everything to be the exact same as it was before, and it can't be right. There's a reason why it's not. It's it, There's a reason why it's antiquated. Right? There's a reason why it's legacy. There's a reason why no one supports it anymore. Um, it's because everyone's gotten away from that type of architecture. Uh, beyond that, you know, doing things. So doing things like you've always done, you know, when everyone says that's how we've always done it, it's, it's my, my least favorite lines ever to hear anyone say in the world. Cuz I'm like, Well, that didn't work for you before, so why would you continue to do it the same way? And then the other thing is, is that especially with NiFi, so Mark Payne, who is, uh, one of the co-creators of NiFi, and obviously a PMC, he put out a series of videos called anti-patterns. So it's five videos where he actually goes and he, he figures out, or tells people like, here are the things that we see in the field that people should never do. So NiFi has evolved a ton over the years, when it first started, everything was about, you know, the flow file or the stream. Now they realize that, it needs to also have a lot of record capabilities. So you see like old-school methodologies or like, you know, if you go to Stack Overflow, it's like, well if you just do this, this and this, this will work. And you're like, Great. That was true like six years ago, but like now that's not the best way to do it. You know, there's a lot that's changed. So like, updating and enriching records. There's literally a processor called update record, whereas before, back in the day, you usually had to then pull it up into an attribute, use the expression language to go ahead and modify the data, then pull it back down. And what people didn't realize is like in order to do that with a record, you have to split the file. So if it's millions of records, you now have gone from one record to millions, they share the attributes across the line except for the ones that you wanna modify. All of that gets stored in a JVM. So, you know, you, you essentially get this memory overload that's happening. So there's just so much like bad patterns that occur when people build flows, and it's, it's primarily because they haven't really been up to speed on the net new stuff. So for me, like, you know, Watching Mark's videos. Like whenever I go to a customer and I see like bad things, I'm like, You need to watch this. Even when I'm just actually there, just getting introduced to NiFi, I'm like, Before you build anything, please watch these videos. Like they're very important videos to watch so you don't make mistakes.
Matt:
28:42
Hmm.
Tim:
28:43
That is critical.
Matt:
28:47
Yeah. Okay. Cool, cool. So not only are there mistakes that people make, they're, they just don't understand the technology sometimes. Yeah. And that's another area that I see quite often, right? So when, when you've got someone who is migrating from a legacy system, you know, and the re-architecture, I, I know it's a big thing. I've seen it myself with other, other technologies as well. Um, but I think it's also just the knowledge gap. So are there a few features or things that are underutilized or under known about like the hidden features or got, Everybody should know this, but they don't. Let me tell everybody what this thing is.
John:
29:25
So through the UI perspective, I, a lot of people don't realize this, but when you click on a processor, when you right click on a processor, there's a view usage category. That view usage category takes you directly to the documentation of that processor.
Matt:
29:40
Hmm.
John:
29:40
And I couldn't tell you how many times, like I, I've worked with a customer and they're like, Well, I don't understand what this means. Right. So now you highlight over it and it usually does give a description, but it literally having that capability to go right to the doc.
Matt:
29:52
Hmm.
John:
29:52
From that specific processor, not having to Google it. . Um, and especially like a guy like, you know, Tim or I have been using NiFi for a very long time, so our cache on our browser might take us back to like, you know, NiFi 1.1, We're on 1.17, about to be 1.18 out.
Tim:
30:09
Oh, 1.18 is out. Man, I'm running.
John:
30:10
Oh, 1.18 is out. There you go. See, I'm a little behind of the time, so I've been traveling, so I apologize. But yeah,
Tim:
30:15
Like two weeks. . John: Well, I've been on the road like the, you know, you look at, you look at that, you know, there's been so many changes that have occurred, like, and those processors change a little bit. Like they're not drastic changes, but there might be net new functionality. Another thing that people don't use a lot, um, is parameters. Um, we are trying to push, push, push, push, push. The utilization of parameters. Back in the day, and this goes back to the what used to be done versus now is variables were the thing. So everyone used variables, but the problem is, is variables, you know, don't have the same richness that a parameter does. So if you're using a variable before at NiFi, you need to use a parameter. If you have things that you wanna port over easy from like one environment to another, use parameters. Because if the parameters set, if you're using parameters, you can move that flow relatively easy, and then point to like maybe a file and system or something. And then essentially, you know, you don't have to change every processor where like you have a URL or you have a path. Right. If that changes on the new system.
Matt:
31:22
Oh, cool. Yeah, I mean, very useful. They, they seem like common sense things, but they're usability of, you know, features like that. They make a lot of difference in people's lives on a day-to-day basis.
John:
31:34
Yeah. Yeah, Completely agree.
Matt:
31:36
So, Tim, final thoughts from you.
Tim:
31:40
Well, I think the, it goes down to, you know, when, when you have someone who's got a lot of experience with it, listen to 'em. Uh, you know, if you have, if you have the pleasure of having someone like John Kuchmek in as, uh, you know, when you're a potential customer or customer or you're at a meetup or an event, you know, listen to what they're saying and, you know, ask for things like we were talking about today, anti-patterns. You know, where, where's the best place to get information. You know, what should I use together and what should I not like? People ask me, should I do everything in Pulsar? No. There're things that go in NiFi. There're things that go in Flink. There're things that go in Spark. If you try to, like, I've got the one thing that does everything well, you know, a Swiss Army knife works, but you, if you built a house with it, you're spending a lot of time and you're suffering. Don't, just because it can do it, you know, a brick, I could use a brick to, uh, tear down a house, but you know, there might be some, uh, I might get in trouble, you know?
John:
32:47
Yeah.
Matt:
32:49
Makes sense. So, John, thanks for hanging out with us today. We appreciate you sharing some knowledge with us and telling us a little bit about NiFi and some of the other experiences you've had in the background here.
John:
33:00
Yeah. Cool. Well, thanks for having me. It's fun. I really appreciate it.
Matt:
33:04
Alrighty. Well, everybody, until next time, don't forget to subscribe, like, and follow us. Uh, we would love that. So, we'll catch you later.
