Crossing The Streams Episode 2 - Frank Kelly[00:00:00]
[00:00:00] Matt Yonkovit: All right. Hello everyone and welcome to a new podcast. Um, as many of you know, I have recently joined Stream Native and I am looking for people in the community who are using Pulsar and other, streaming products to talk to, to learn more about it. And today I have a special guest. I have Frank Kelly from Cogito.
Frank, how you doing?
[00:00:26] Frank Kelly: I'm excellent. How are.
[00:00:27] Matt Yonkovit: I am Great. I'm great. So Frank, , I know that Cogito uses, Pulsar, but I was hoping maybe we could get a little bit more background on who you are, Frank, how did you get involved in the streaming messaging space? Tell us a little bit about your background.
[00:00:43] Frank Kelly: Yeah. My streaming messaging background goes back quite a way. So, uh, back in the old days of your, I started off playing around with, uh, IBM MQ series, uh, and that. Experience had gave me a lot of trouble of like lost messages and what happens if I can't connect to the [00:01:00] queue. And all these kind of like these challenges and over time, future companies, I've basically worked with, uh, active mq J Boss, mq, always in a high scale kind of environment, either finance, telecommunications, um, social media, that kind of world.
Um, eventually ended up being the SQS, sns and uh, Apache Pulsar has kind of been. The the latest and greatest thing I've been using.
[00:01:24] Matt Yonkovit: So when you, when we talk about messaging and you. For, we have lots of different people who watch this. We're talking about, the ability to send data to multiple applications, to share data across different, uh, systems. Maybe if you could give a couple of examples of things you've done in the past from a messaging perspective, just to give people some context what that looks.
[00:01:48] Frank Kelly: Yeah, it looked, it looked very different. So there's kind of, um, back in the JMS days, they talked about point to point and then Pub/SUB, right? So point to point is, you know, I'm, I'm putting messages into a queue for a single consumer that's going to, [00:02:00] to read from it. So it could be, you know, back in the days we're doing stock price changes when we could actually keep up enough.
So we put stock price changes into a queue and then something would be reading from that and try to update a. So that's kind of a point to point, um, messaging. and Pub/Sub is kind of a, you know, these things are happening , a number of people can find out about it. So again, in that kind of financial world, um, there we use Tibco, right?
We broadcast on these topics and that all these different things could be listening to it, something, it might be updating the user interface in real time. Something might be trying to update a database, uh, something might be doing some analytics. And uh, so that's another. Messaging use case. And then of course, then there's the whole streaming use case, which, you know, we also use with Pulsar as well for, for our case, which is new to me.
Um, which is basically there's a continuous stream of binary information and that's a whole separate kind of use case,
[00:02:52] Matt Yonkovit: Yeah. So instead of just one message, fire it off, be done Now. It's always, always on, always coming in.
[00:02:59] Frank Kelly: The [00:03:00] challenges around that I go, you know, especially if it's a binary stream, not something structured like Js O. You can't afford to lose a message and you can't have duplication cuz then the binary becomes unreadable, it becomes garbled. And um, that's one particular area where for us, Paul sars really shined, is that we really, I can't think of a single case in the last two years where we lost a message or had a duplication.
It just works.
[00:03:24] Matt Yonkovit: Well, that leads to the question, you know, working for Cogito. Maybe for those who don't know who Cogito, what Cogito does, can you tell us a little bit about Cogito and how you're using Pulsar.
[00:03:33] Frank Kelly: Yep. We are a 12 year old startup, uh, originally out of mit, and we build, uh, emotion and conversational AI to build a contact center solution. So we're listening in basically on, on contact center agent phone calls, and then giving them behavioral. Uh, nudges and suggestions, um, at different points in the, in the phone conversation.
It could be as simple as, you know, you're being very quiet for a long time. You know, often [00:04:00] agents are often typing stuff into a computer and they forget the fact that they're on the phone and it's all gone quiet. Um, it could be something as picking up, you know, the, the customer is clearly very upset right now, so, you know, we can give them, um, advice on, you know, uh, tailored advice on how to, how to handle that at the moment.
um, we can, detective people are talking over each other and so we kind of do that to build up and, and not kind of a, you're a bad agent type of deal, but kind of a, a friendly guide to the agent to help them. We find that agents are actually quite good at their, their jobs. This is about giving them little extra boost and also giving the supervisors insight.
So they're managing tens of agents. How are they? Across, you know, multiple business lines, different goals, different teams, and then executives look down across everybody. So how are things going? Are we getting, you know, are we improving our, our behaviors? Um, and um, ultimately there's the, the net goal is, uh, cost [00:05:00] improvements really for the, for the, uh, for the company.
Also working on their brand. Um, you know, so we, we work with, you know, uh, telecommunications companies, healthcare companies, finance, c. Companies that are very protective of their brand or who have a very complex relationship with people cuz the, the laws are complex and so there's often complex and very weighty conversations going
[00:05:21] Matt Yonkovit: So you have someone who's talking like we are. So let, let's use this as an example. So, we're recording audio. Um, we're actually doing this live right now, and it's going up to the server that we're using Riverside. As we're we're recording, Cogito basically would be then sitting here and listening in and being able to take this live as this is streaming in and be able to say, you know, Oh, Matt, you haven't said anything in 30 seconds.
You, you should probably ask another question.
[00:05:48] Frank Kelly: Yeah. Yeah. So I mean, this is the, the really interesting part is that we have very high response time s. High, very low responsive SLAs. They're high, high bar to meet. Um, and so we're basically trying to get the audio, [00:06:00] um, send it over the wire, send it to our compute and our AI back, uh, backend systems, have them process it, and AI is very competition intensive.
Um, and then take pull out events, send those back to Pulsar and send them up to the user interface all within 250 milliseconds. And
[00:06:15] Matt Yonkovit: Yeah, and, and that's the 250 milliseconds. That's. Nothing to sneeze at. I mean, when we talk about millisecond times, those are difficult, especially when you're talking about cross the wire. You're going to cross data centers. Um, you're going to potentially individual, you know, machines, uh, at people, sometimes working from home.
[00:06:33] Frank Kelly: Yeah, that, that time certainly adds up. And, uh, you know, one of the other challenges is we're looking now at, contact centers that have thousands, tens of thousands of registered agents of whom. Five, six, 7,000 could be online at the same time. So we are doing high response time and massive throughput as well.
So it's,
[00:06:51] Matt Yonkovit: Yeah. That's gotta be a lot of throughput because when we talk about like processing text, text can be big, but audio is bigger. So what, what sort of sizes are [00:07:00] we talking about?
[00:07:01] Frank Kelly: So we, we we're currently doing audio, It's around 32, uh, kilobytes per second, which is not a lot. But again, you multiply that by thousands of people. You're talking easily gigabits per second. Um, that kinda speed. And thankfully that's areas for, plus our compression is a lifesaver and we can compress it on the, the clients side with very little overhead and it just, you know, transmit it through.
But we're willing to pay for the, we're always willing to pay for the CPU cuz the IO really is, uh, expensive.
[00:07:28] Matt Yonkovit: Well, and that's something that, Pulsar does have a pretty good model of elasticity, right? So the ability to spin up additional workers and consume, CPU differently because pulsars architecture's a little different than other messaging systems where. The compute storage is broken up.
[00:07:46] Frank Kelly: It is. Yeah,
[00:07:47] Matt Yonkovit: yeah. And, and, and so you are, are you spinning up additional workers as the workload increases? So if you have 5,000 people online now, and then all of a sudden it jumps to 10, then you just start spinning up additionals.
[00:07:58] Frank Kelly: we, we want to, we [00:08:00] sort of want to get there. We're actually testing it now with the Pulsar, our auto scaling, low balancing. . One of the things we often find is, it's often said that the, the brokers are stateless. And that's really, it's really not true. And we find that as we, these name space bundles.
So, you know, pulsar breaks a, uh, a name space of, a bunch of topics into Shards, if you will also call name space bundles. And those name space bundles are assigned to brokers. And those things will basically, as you add more brokers, they have to move around, Okay. Balance that load. And in that moment they're moving.
We could have active topic. Talking to one broker and all of a sudden now they've gotta talk to another broker. And so what we're finding is the clients aren't, The clients are able to recover, but not able to recover fast enough for our use cases. So we're kinda working with the folks at StreamNative to, to kind of optimize that and, um, you know, do detect things faster, fail over more quickly.
[00:08:50] Matt Yonkovit: Okay. Yeah, and I think that, that's not a unique problem because as you scale any sort of messaging system, you have similar, um, challenges. And I think [00:09:00] architecturally, Pulsar allows a little bit more flexibility there. It sounds like there's some work that needs to get done. When you're talking about the older systems , whether it's an active MQ or you know, Rabbit in some cases, or even a Kafka, there might be other limitations that are exposed that are a little bit different.
So I think that as we start to scale up, we start to run into those, really deeper problems at mass scale and, you're talking tens of gigs of data a second potentially on your side.
[00:09:30] Frank Kelly: Yeah. Yeah. People often talk about, you know, scaling is easy and a lot of the solutions out there are really about stainless scaling. So you've got servers that are stainless and talking to databases where the state is. But scaling stateful storage has been, at least in my experience, is extremely hard.
And so it's, you know, whether that's relational databases or to, you know, high end key value stores like Dynamo and stuff like that, um, you know, it's very hard to get it right and you really. A lot of, um, a lot of investment to get that optimized for, for lot of number of [00:10:00] different use cases.
[00:10:01] Matt Yonkovit: So were you at Cogito when they implemented P Star? Initially.
[00:10:05] Frank Kelly: Uh, yeah. I was out there from the, from the beginning. I wasn't there at the moment they was chosen. But, um, you know, the, the criteria that we use to choose P Star really, first of all, multitenancy support, um, which will be great for us in, in terms of cost savings, the encryption authorization built in, which is fantastic.
Um, you know, the scalability story that they had, um, was really awesome. And, um, the support for all those clients. So, you know, we wanted to go from a kind of monolith to a microservice architecture where we wanted to be agnostic about clients. We didn't wanna spend all our time writing clients for something maybe we wanted to have.
And so having the Java go and Python clients ready to go was, you know, really important for.
[00:10:46] Matt Yonkovit: Okay, were, were you, So since it was a startup, you didn't have a lot of legacy code though, that you had to worry about either rewriting, did you?
[00:10:53] Frank Kelly: Well, we did, I mean the, the challenge, uh, was really that we had a number of customers on our, uh, platform, one as we [00:11:00] called it, and we wanted to migrate them to platform two. And so we were basically, if you know the story, but the big dig in Boston, right? So, you know that there's, that probably lot people to know.
It's basically, we used to have this, you know, raised highway going through Boston and decided that's replaced it with a. And so they had to basic dig this massive tunnel under it while keeping the thing running on top. And that's kind of what we did. We basically pulled a bunch of interfaces in front, kept the user interface working against Platform one and platform two, and then basically, you know, implement everything exactly the same underneath the covers with some switches.
And, and so that was, um, pretty interesting and pretty challenging. But we're, we've been success.
[00:11:37] Matt Yonkovit: Now you talked about multi-tenancy. For a lot of people, multi-tenancy, um, if they've got one app, they might not think about it. Or even if they have multiple apps that need data, they're not necessarily thinking about the segregation. You mentioned cost savings there as a potential reason.
Maybe talk a little bit about that multi-tenancy and why that was attractive.
[00:11:56] Frank Kelly: Yeah. I mean for, we have a number of different customer [00:12:00] bases out there. Some are are larger, some are smaller. uh, yeah. So for some of them, again, with any environment, we're setting up databases. We're setting up Pulsar with its zookeepers and its bookies, and we're always saying, Well, for high availability.
We want, you know, at least two broker pods and we need a quorum in Booky. So I need three Booky pods. I need, all of a sudden I've got nine pods, I've got three zookeepers, three bookies, two bro brokers, and two proxy. So maybe that's 10. Right? So I've got 10 pods right off the the bat, and I need them be pretty snappy and responsive.
So giving them some CPU shares this memory. And if that's for a customer base where we wanna get like 50 or 60 customers online, that's even, that's pretty, you know, expensive to run. So we're trying to do is basically put clients together. They're logically separated, logical that the exact term logical tenant safety, if you will, in that environment.
So they're in separate, in the same Kubernetes name space, but different Pulsar, our name spaces with other controls. And, [00:13:00] uh, we can use that to save money basically so that those 10 pods can work for like five customers rather than just one.
[00:13:05] Matt Yonkovit: Yeah, so operationally it's more efficient because there's less to manage, but also, you know, you've got the, ability to use less, hardware and less components. So. Okay. That totally makes sense. You know, part of that is also the security of it, making sure that the data is segregated, which is nice to have that capability built in.
Cause that's an absolute requirement, um, as well. Yeah.
[00:13:29] Frank Kelly: So security was always key. So like, you know, all our ABS drives are encrypted. We're always making sure that, you know, we have proper controls, authorization, authentication within our spring applications and elsewhere. Um, we were, one of the good things we did, we use Isto is our service me, and we're able to bolt that on with Pulsar to make sure that, you know, there's some pretty strong controls around it.
I. PSR has his own authentication authorization mechanism, but we just didn't fancy spending a lot of time injecting certs and doing mtls when Isto will kind of do that for us, and it's been [00:14:00] pretty success.
[00:14:01] Matt Yonkovit: Oh, cool. Yeah. So you are, you have a team of developers who are using this and they're building on it and they're enhancing it. Um, maybe tell me a little bit about, like from your role, when you have developers who are coming on, what are the things that they find. Either interesting or odd about Pulsar.
Are there those things that you have to kind of explain the topics or explain, you know, what's going on?
[00:14:22] Frank Kelly: Subscriptions. That's what I find is the first thing. So I remember my first. Playing with pulse hour. All right. I created a topic, I wrote a message. I created a consumer. I listened to the topic end, nothing. What? I'm like, I'm used to cues. I write something to a queue, it stays there. I read from the queue, its, it's mine.
I'm like, What? I'm like, Spend hours. I'm like, What? So that's the big difference, right? Is that if the subscription wasn't preexisting, then. You're not gonna get the message cuz they're default for Java, at least I think for is to get latest message. So the minute you start, if I send a message at Time zero, I try to consume a time [00:15:00] one, it says give all message from time one forward.
And so you basically have to tell it all right, you gotta go back to earliest and get all the messages. And that's when I start learning about subscriptions. And then, you know, there's these things called backlogs, which sound like they're a queue backlog. There sound like there were a bunch of stuff in a queue, but they're not.
They're just like a logical marker. So that, that's pretty confusing to people, and that that part is the first part to get over is, is, um, what does the backlog mean? What does subscriptions mean? Um, you know, from a operational standpoint, the subscriptions are a little bit challenging too because other people can create subscriptions that are durable, they last forever, and if they last forever, you will never delete that data, right?
That, that old subscriptions hangs around Then. That, you know, pollster says, Well, if they haven't caught up, I'm just gonna hold onto that data just in case they come on, you know, on Tuesday, decide to catch up. But it's like it's a problem for retention. Cause all of a sudden I'm paying, well, [00:16:00] pretty dollar for all this EBS drives.
I'm like, where are all my disc space going? It's like, okay, we gotta got rogue subscription out there and gonna deleted.
[00:16:08] Matt Yonkovit: Hmm. I think anytime you change how you've thought about building an application or how a piece of technology works, it's often difficult to like make the transition because you just fall into old habits.
[00:16:18] Frank Kelly: That's right. Totally. And so you start thinking it's more, more of a streaming world of, you know, there's multiple people at different points in the queue, if you will, in the topic. Then it becomes a little bit easier to understand than.
[00:16:29] Matt Yonkovit: We were chatting, in a pre-call about, one of those other things, which is like the negative acknowledgement and how, developers are like, What about my exception handling?
[00:16:39] Frank Kelly: Right. Yeah. I mean, exceptionally is is always hard. I used to do this with MQ series back in the day, and it. I got a message, I can't handle it. I can't write the database. Oh, what do I do? And that's what we were doing is like write into a dead letter queue that somebody else pick it up. I'm like, Okay, now I've just created another software development problem and I've gotta reach for another queue and figure out how to handle it.
And what if somebody else already picked it up? And so the nice thing is [00:17:00] PTR gives this knack ability, which is if I can't handle just, you can either not acknowledge it, then a timer goes. In which case the message will be pre delivered or I can negatively acknowledge that I can handle this right now, somebody else can go handle it.
And so you can kind of basically say there's some errors that you can retry on, some errors, you can't. And so the negative acknowledgement lets you handle that, which is kind of nice because there there's times where we're trying to basically, um, you know, maybe offload data to s3 and, uh, you know, something happened For whatever reason, we couldn't, you know, the database was slammed and we couldn't get where the location was.
Retried 15 times, which we have configured. It's like, you know, every 10 seconds. Then, you know, if it doesn't work well, you know, then we have an alert. But, having to automatic the retry is automatic. Give you kind of a lot of flexibility.
[00:17:50] Matt Yonkovit: Oh, that's cool. Yeah. One of the other things that, you know, I've been reading a lot about is, the high availability, set up and the geographic replication available in Pulser [00:18:00] are, are, you know, I'm assuming that there is some, you know, geo specific things that you're doing.
and you are potentially taking use of that. So maybe tell us a little bit, about what that high availability looks like for, for you all and how you achieve that, especially at the throughput that you have. Um, you know, that, that's gotta be a, um, an interesting recovery time.
[00:18:22] Frank Kelly: Yeah, that's a great question. We really tend to use geographic replication. So we we're often, we just deploy really in the AWS region that's closest to our, to the customer. Uh, so it could be US East, S one, that kind of stuff. Um, but in terms of ha, I mean, we're always trying to be aware of, you know, Zuki has its own ha, right?
So it's gotta be three pods, but then the question is how are they spread out across nodes in Kubernetes? That across availability zones. So the, you know, the P star chart, the P star home chart has a bunch of anti affinity stuff, so it's certainly spreading things out across nodes. Um, and then, you know, one of things we have to still kind of go back and work on is, [00:19:00] um, is that also spread out across availability zones?
And my guess is it's not, and uh, you know, we, we've definitely the, that someone to work on for zookeeper and certainly on the bookies as well. Uh, it's even probably more critical, but the fact. If the zookeeper fails and bookings don't, it might as well be the same thing because we found that, you know, once those two things get outta sync, we're kind of, we're kind of hose, um, brokers are a little, they're stateful with a little more recoverable.
Um, so yeah, I think the HA is really having multiple pots spread across node. But the area to work on is, is the, um, the anti affinity. So breaking it out by availability zones.
[00:19:40] Matt Yonkovit: Okay. Well, and so you've been working with Pulsar for a while now. What's your favorite feature?
[00:19:46] Frank Kelly: What's my favorite feature? Um,
[00:19:51] Matt Yonkovit: Oh, there.
[00:19:52] Frank Kelly: I was, yeah, there's so many good things. So
just last night I was, , did [00:20:00] a production change and I just turned it on and I listened to it, to the, the audio streams, this log as a user and got listened to phone, phone calls going on. It's just seamless, right? So we're basically listening to the audio stream chunky in a, in the separate messages.
You know, it's kinda like Willy Wonka, that Mike TV moment. Do you remember that one? You like rips the kid apart, put some, opens it, and we're pull, pulling it all back together again. But it's, it's seamless, right? And it's just listening to it and it's like, I can't believe that's eight messages a second.
And it's a seamless, There's no, there's no glitches, There's no, um, you know,
[00:20:36] Matt Yonkovit: it all works together then really is
[00:20:37] Frank Kelly: just works. Yeah. And so that's working with compression and authentication and all behind the scenes going across, you know, uh, multiple VPCs and, you know, um, to into memory and onto disk and then back out. And then it's kinda like, it's pretty cool.
It's pretty cool. It just works. That part just works. The compression is awesome too. I mean, that, we just turned that on and we got like a three x [00:21:00] improvement in, in a lot of our performance benchmarks and IO and, and,
[00:21:04] Matt Yonkovit: Oh really? Wow. That's, That's a pretty big savings.
[00:21:07] Frank Kelly: I guess web isn't as, isn't as, uh, efficient as one would things. There's certainly other, um, audio formats that we're looking at that would be, you know, compressed by default.
But it was kinda nice to get that out the box. Cause we were looking at staring down 500 megabytes per second. Then all of a sudden we just went to 150. It was like, yeah. Sweet. That's much
[00:21:25] Matt Yonkovit: Well, yeah, that, that, that does, that does reduce the costs and
[00:21:27] Frank Kelly: It does.
[00:21:28] Matt Yonkovit: the bandwidth there. Um, so as people are deploying Pulser and, and as your users or developers are starting to use it, Is there a common like problem that continually comes up, like, you know, an issue that you see over and over again?
[00:21:42] Frank Kelly: Common problem.
I don't chose a common problem, but there's a common piece of advice, which is Pulser, uh, exposes a whole bunch of metrics for Prometheus and that, um, you really need to graph those.
[00:21:59] Matt Yonkovit: [00:22:00] Ah.
[00:22:00] Frank Kelly: for the proxy, the broker, the bookies, the zookeeper. That will be your lifesaver. I mean, for us, we're kind of a somewhat of a unique use case in the sense that we're doing all these binary streams and stuff like that.
So for us, we're kind of really monitoring broker, uh, ledger, CAEs and um, you know, that kind of stuff. But in terms of, you know, stiff goes wrong, uh, we had an issue this morning, right? You know, we, all of a sudden our booky response times went through really high. And so we perme these alerting kind of goes.
You're Booky alert, you're booky response time. P 99. Just hit 30 seconds. I'm like, Okay, . That's not good. That's not good. What the heck? So we looked at it and was like, it was a big read load. Where'd that reload come from? Right? And then having those tools to track that down, cuz between the proxies, the brokers, the bookies, the zookeepers, your, your drives, all the pods, debugging is gonna be hard.
And so, you know, upfront, invest that time. You know, whether it's your pana charts, uh, alerting, [00:23:00] know what goes wrong and alert on it and be ready. Uh, cuz that's, that's the hard part you, because, and we also wrote run books, right? It's like, hey, if you see this, Lord, look here cuz you're gonna get it at three o'clock in the morning and you're kinda gonna go, uh, I'm not caffeinated.
What do I do? Like, okay. I wrote myself a room book. Okay, Dear Frank, here's what you're supposed to do. Like, thanks, so,
[00:23:21] Matt Yonkovit: Frank, I know you were, you were gonna speak at the Pulsar Summit, but you couldn't make it this
[00:23:24] Frank Kelly: I know that
[00:23:25] Matt Yonkovit: was the topic you were gonna give?
[00:23:27] Frank Kelly: So it was basically, I gave a topic the prior year and uh, it was more of the, more of a things we discovered. So first year was all about, we discovered to, um, so this is another little thing. They always say, Well, you know, we, we write, we write to the journal, but you read from the ledger, right?
And it's like, and the two never really meet and like, well, they have to, Right? The journal has to. Written to the ledger. And so we learned that if you're reading for the ledger at the same time as you're trying to write to the journal, write as the journal's trying to write to the ledger you're trying to read for the ledger, you can basically jam up the bookies.
[00:24:00] And so what we learned was, again, we're in this for, for unfortunate real time case. If we've made the broker cash is really big, then we would just read for the broker cash and never hit the bookies. And then all of a sudden our throughput was off the races. So that was one big positive from 20 21, 20 22 was when I realized that.
We would be creating a lot of topics, um, you know, very, very quickly. And, uh, for that we, you know, we needed to improve zookeeper and a lot of response times there around zookeeper and scaling zookeeper. And so zookeeper has a number of settings, um, so of which are just, you know, now available to improve that scalability.
So what that metadata, zookeepers, and metadata store keeps track of all the topics. And as they're created and deleted, it gets quite a bit of. And so that was a key there to, to, to monitor that.
[00:24:49] Matt Yonkovit: Yeah, so I mean, it's just really about the scale. Now you mentioned like you're creating a lot of topics because you create a topic per phone.
[00:24:56] Frank Kelly: Yeah. So for every phone call, those two topics, there's one which is the [00:25:00] audio and one which is the, um, the, the, the analytics. And we had multiple debates about this, about what the way to go. And we wanted to do maybe shared subscriptions, a big fat topics. And that was kind of suggestion. But what we realized with the, the kind big fat topics, Our, our AI algorithms are kind of, you know, small little pods, thousands of them, thousands and thousands of pods, and they were just like, I want one audio stream and I'll produce one, one, uh, analytics stream.
Great. The problem is if we have one big fat pipe, um, that would, that fat pipe of IO would have to go all to the, um, that single pod and it would only have to pick like one 1000 to one 5000th of that io. It would be interested. . And so, um, I know those talking future for doing some, uh, filtering basically, uh, on the server side, but we felt it was just easier to manage and get go live.
We had, you know, one topic per audio stream and then, you know, um, it worked out pretty well. But there, [00:26:00] it's not the default set up for Pulser to think about that. So there was a number. Settings we had to tweak. I can find my, my chalk from 2021 about that. Um, but you know, as, as with everything in Pollstar, there's a setting for it.
You just have to find out what it is.
[00:26:16] Matt Yonkovit: Yeah. No, and it's, it's interesting because. You know, it, it goes to show you, you know, as you're building out applications, uh, especially as you grow and scale, oftentimes you need to think about the workload and optimize, you know, the infrastructure for the workload, not just on the setting side, but in your, your, your case.
You're like, Oh, well one topic, per phone call makes a lot of sense for us. That's not something. You would necessarily think about, but I think that as we start to think about like the future of, you know, how things grow, how things evolve, those are the types of design decisions that a lot of people might not realize they need to make early on or as early as possible.
[00:26:59] Frank Kelly: [00:27:00] yeah, yeah. I was just thinking today, just like, you know, if there's a lesson I'd learned in 20 plus years, it was like, don't front your database with a queue, cuz you, you can jam messages into your queue faster than you can jam them into the database, those kind of things. But you really have to be aware of rough order of magnitude of, of each thing.
Right? So databases you're talking about, say a 10 millisecond insert. Maybe now it's locally or maybe it's not, you know, me cash might, you know, it's one millisecond, whatever, but that'd be different. But then you gotta understand the order of magnitude of things and we can jam tons of messages into P Star.
We're like looking at, we have record everything. We're like, Oh, P star send times, like less than a millisecond. I'm like, Yeah, you can really, you know, overwhelm things downstream if you want to with, uh, with Pulsar as the front end. So you really be aware of proper choices of storage, um, and not just for.
Scale, but also cost, right? So we were trying to jam stuff into, um, our bookies, and then we realized, Oh, let's just pick [00:28:00] GP three drives. They're really, really fast and you know, they're inexpensive compared to iops, optimized the I two s and, uh, you know, was expensive. We still paying a lot of money, and we realized, well, if we just add more bookies and had GP two achieve the same scale with uh, less cost.
And, but yeah, you have to really be thinking. The relative order of magnitude of those, those bits. Um, again, if you're just sending a couple messages around, you know, every couple of seconds, you're not too worried about it. But if you're really at the high scale world, um, yeah, you're gonna be living and breathing
[00:28:31] Matt Yonkovit: There's a fine line between over or early optimization or future proofing, like spending too much time there and, and not spending enough time. Uh, and that's a whole interesting, you know, debate and discussion, but
[00:28:42] Frank Kelly: I think layering, make sure you have the proper interfaces so you can swap out implementations is always important. That's one of the things we were also gonna do as well, is we have our, our telephony systems and our compute systems are directly tied into Pulsar with the library that we wrote. And now we're like, You know what?
We wanna pull that out and abstract that away just because. , [00:29:00] um, you know, we're often like, Oh, we gotta change the library and add this little setting. NA's gotta go to our, you know, telephony team. And then they have to deploy it and they've gotta make sure that's deployed everywhere. And that's the thing we have to organize.
It's like, well, we had an interface in front that was abstracted away then, you know, that, that would be a lot easier. So,
[00:29:18] Matt Yonkovit: Oh, very
[00:29:19] Frank Kelly: yeah. Well, a lot of lessons learned. Good.
[00:29:21] Matt Yonkovit: Ah, well, Frank, I wanna thank you for, spending some time with me and, telling me about how you're using Pulsar. Um, helping me learn a little bit more about, how people are deploying it and using it, and some of those things we gotta watch out for. I really do appreciate it.
[00:29:36] Frank Kelly: Not all. Thank you very much. It was an absolute pleasure.
[00:00:00] Matt Yonkovit: All right. Hello everyone and welcome to a new podcast. Um, as many of you know, I have recently joined Stream Native and I am looking for people in the community who are using Pulsar and other, streaming products to talk to, to learn more about it. And today I have a special guest. I have Frank Kelly from Cogito.
Frank, how you doing?
[00:00:26] Frank Kelly: I'm excellent. How are.
[00:00:27] Matt Yonkovit: I am Great. I'm great. So Frank, , I know that Cogito uses, Pulsar, but I was hoping maybe we could get a little bit more background on who you are, Frank, how did you get involved in the streaming messaging space? Tell us a little bit about your background.
[00:00:43] Frank Kelly: Yeah. My streaming messaging background goes back quite a way. So, uh, back in the old days of your, I started off playing around with, uh, IBM MQ series, uh, and that. Experience had gave me a lot of trouble of like lost messages and what happens if I can't connect to the [00:01:00] queue. And all these kind of like these challenges and over time, future companies, I've basically worked with, uh, active mq J Boss, mq, always in a high scale kind of environment, either finance, telecommunications, um, social media, that kind of world.
Um, eventually ended up being the SQS, sns and uh, Apache Pulsar has kind of been. The the latest and greatest thing I've been using.
[00:01:24] Matt Yonkovit: So when you, when we talk about messaging and you. For, we have lots of different people who watch this. We're talking about, the ability to send data to multiple applications, to share data across different, uh, systems. Maybe if you could give a couple of examples of things you've done in the past from a messaging perspective, just to give people some context what that looks.
[00:01:48] Frank Kelly: Yeah, it looked, it looked very different. So there's kind of, um, back in the JMS days, they talked about point to point and then Pub/SUB, right? So point to point is, you know, I'm, I'm putting messages into a queue for a single consumer that's going to, [00:02:00] to read from it. So it could be, you know, back in the days we're doing stock price changes when we could actually keep up enough.
So we put stock price changes into a queue and then something would be reading from that and try to update a. So that's kind of a point to point, um, messaging. and Pub/Sub is kind of a, you know, these things are happening , a number of people can find out about it. So again, in that kind of financial world, um, there we use Tibco, right?
We broadcast on these topics and that all these different things could be listening to it, something, it might be updating the user interface in real time. Something might be trying to update a database, uh, something might be doing some analytics. And uh, so that's another. Messaging use case. And then of course, then there's the whole streaming use case, which, you know, we also use with Pulsar as well for, for our case, which is new to me.
Um, which is basically there's a continuous stream of binary information and that's a whole separate kind of use case,
[00:02:52] Matt Yonkovit: Yeah. So instead of just one message, fire it off, be done Now. It's always, always on, always coming in.
[00:02:59] Frank Kelly: The [00:03:00] challenges around that I go, you know, especially if it's a binary stream, not something structured like Js O. You can't afford to lose a message and you can't have duplication cuz then the binary becomes unreadable, it becomes garbled. And um, that's one particular area where for us, Paul sars really shined, is that we really, I can't think of a single case in the last two years where we lost a message or had a duplication.
It just works.
[00:03:24] Matt Yonkovit: Well, that leads to the question, you know, working for Cogito. Maybe for those who don't know who Cogito, what Cogito does, can you tell us a little bit about Cogito and how you're using Pulsar.
[00:03:33] Frank Kelly: Yep. We are a 12 year old startup, uh, originally out of mit, and we build, uh, emotion and conversational AI to build a contact center solution. So we're listening in basically on, on contact center agent phone calls, and then giving them behavioral. Uh, nudges and suggestions, um, at different points in the, in the phone conversation.
It could be as simple as, you know, you're being very quiet for a long time. You know, often [00:04:00] agents are often typing stuff into a computer and they forget the fact that they're on the phone and it's all gone quiet. Um, it could be something as picking up, you know, the, the customer is clearly very upset right now, so, you know, we can give them, um, advice on, you know, uh, tailored advice on how to, how to handle that at the moment.
um, we can, detective people are talking over each other and so we kind of do that to build up and, and not kind of a, you're a bad agent type of deal, but kind of a, a friendly guide to the agent to help them. We find that agents are actually quite good at their, their jobs. This is about giving them little extra boost and also giving the supervisors insight.
So they're managing tens of agents. How are they? Across, you know, multiple business lines, different goals, different teams, and then executives look down across everybody. So how are things going? Are we getting, you know, are we improving our, our behaviors? Um, and um, ultimately there's the, the net goal is, uh, cost [00:05:00] improvements really for the, for the, uh, for the company.
Also working on their brand. Um, you know, so we, we work with, you know, uh, telecommunications companies, healthcare companies, finance, c. Companies that are very protective of their brand or who have a very complex relationship with people cuz the, the laws are complex and so there's often complex and very weighty conversations going
[00:05:21] Matt Yonkovit: So you have someone who's talking like we are. So let, let's use this as an example. So, we're recording audio. Um, we're actually doing this live right now, and it's going up to the server that we're using Riverside. As we're we're recording, Cogito basically would be then sitting here and listening in and being able to take this live as this is streaming in and be able to say, you know, Oh, Matt, you haven't said anything in 30 seconds.
You, you should probably ask another question.
[00:05:48] Frank Kelly: Yeah. Yeah. So I mean, this is the, the really interesting part is that we have very high response time s. High, very low responsive SLAs. They're high, high bar to meet. Um, and so we're basically trying to get the audio, [00:06:00] um, send it over the wire, send it to our compute and our AI back, uh, backend systems, have them process it, and AI is very competition intensive.
Um, and then take pull out events, send those back to Pulsar and send them up to the user interface all within 250 milliseconds. And
[00:06:15] Matt Yonkovit: Yeah, and, and that's the 250 milliseconds. That's. Nothing to sneeze at. I mean, when we talk about millisecond times, those are difficult, especially when you're talking about cross the wire. You're going to cross data centers. Um, you're going to potentially individual, you know, machines, uh, at people, sometimes working from home.
[00:06:33] Frank Kelly: Yeah, that, that time certainly adds up. And, uh, you know, one of the other challenges is we're looking now at, contact centers that have thousands, tens of thousands of registered agents of whom. Five, six, 7,000 could be online at the same time. So we are doing high response time and massive throughput as well.
So it's,
[00:06:51] Matt Yonkovit: Yeah. That's gotta be a lot of throughput because when we talk about like processing text, text can be big, but audio is bigger. So what, what sort of sizes are [00:07:00] we talking about?
[00:07:01] Frank Kelly: So we, we we're currently doing audio, It's around 32, uh, kilobytes per second, which is not a lot. But again, you multiply that by thousands of people. You're talking easily gigabits per second. Um, that kinda speed. And thankfully that's areas for, plus our compression is a lifesaver and we can compress it on the, the clients side with very little overhead and it just, you know, transmit it through.
But we're willing to pay for the, we're always willing to pay for the CPU cuz the IO really is, uh, expensive.
[00:07:28] Matt Yonkovit: Well, and that's something that, Pulsar does have a pretty good model of elasticity, right? So the ability to spin up additional workers and consume, CPU differently because pulsars architecture's a little different than other messaging systems where. The compute storage is broken up.
[00:07:46] Frank Kelly: It is. Yeah,
[00:07:47] Matt Yonkovit: yeah. And, and, and so you are, are you spinning up additional workers as the workload increases? So if you have 5,000 people online now, and then all of a sudden it jumps to 10, then you just start spinning up additionals.
[00:07:58] Frank Kelly: we, we want to, we [00:08:00] sort of want to get there. We're actually testing it now with the Pulsar, our auto scaling, low balancing. . One of the things we often find is, it's often said that the, the brokers are stateless. And that's really, it's really not true. And we find that as we, these name space bundles.
So, you know, pulsar breaks a, uh, a name space of, a bunch of topics into Shards, if you will also call name space bundles. And those name space bundles are assigned to brokers. And those things will basically, as you add more brokers, they have to move around, Okay. Balance that load. And in that moment they're moving.
We could have active topic. Talking to one broker and all of a sudden now they've gotta talk to another broker. And so what we're finding is the clients aren't, The clients are able to recover, but not able to recover fast enough for our use cases. So we're kinda working with the folks at StreamNative to, to kind of optimize that and, um, you know, do detect things faster, fail over more quickly.
[00:08:50] Matt Yonkovit: Okay. Yeah, and I think that, that's not a unique problem because as you scale any sort of messaging system, you have similar, um, challenges. And I think [00:09:00] architecturally, Pulsar allows a little bit more flexibility there. It sounds like there's some work that needs to get done. When you're talking about the older systems , whether it's an active MQ or you know, Rabbit in some cases, or even a Kafka, there might be other limitations that are exposed that are a little bit different.
So I think that as we start to scale up, we start to run into those, really deeper problems at mass scale and, you're talking tens of gigs of data a second potentially on your side.
[00:09:30] Frank Kelly: Yeah. Yeah. People often talk about, you know, scaling is easy and a lot of the solutions out there are really about stainless scaling. So you've got servers that are stainless and talking to databases where the state is. But scaling stateful storage has been, at least in my experience, is extremely hard.
And so it's, you know, whether that's relational databases or to, you know, high end key value stores like Dynamo and stuff like that, um, you know, it's very hard to get it right and you really. A lot of, um, a lot of investment to get that optimized for, for lot of number of [00:10:00] different use cases.
[00:10:01] Matt Yonkovit: So were you at Cogito when they implemented P Star? Initially.
[00:10:05] Frank Kelly: Uh, yeah. I was out there from the, from the beginning. I wasn't there at the moment they was chosen. But, um, you know, the, the criteria that we use to choose P Star really, first of all, multitenancy support, um, which will be great for us in, in terms of cost savings, the encryption authorization built in, which is fantastic.
Um, you know, the scalability story that they had, um, was really awesome. And, um, the support for all those clients. So, you know, we wanted to go from a kind of monolith to a microservice architecture where we wanted to be agnostic about clients. We didn't wanna spend all our time writing clients for something maybe we wanted to have.
And so having the Java go and Python clients ready to go was, you know, really important for.
[00:10:46] Matt Yonkovit: Okay, were, were you, So since it was a startup, you didn't have a lot of legacy code though, that you had to worry about either rewriting, did you?
[00:10:53] Frank Kelly: Well, we did, I mean the, the challenge, uh, was really that we had a number of customers on our, uh, platform, one as we [00:11:00] called it, and we wanted to migrate them to platform two. And so we were basically, if you know the story, but the big dig in Boston, right? So, you know that there's, that probably lot people to know.
It's basically, we used to have this, you know, raised highway going through Boston and decided that's replaced it with a. And so they had to basic dig this massive tunnel under it while keeping the thing running on top. And that's kind of what we did. We basically pulled a bunch of interfaces in front, kept the user interface working against Platform one and platform two, and then basically, you know, implement everything exactly the same underneath the covers with some switches.
And, and so that was, um, pretty interesting and pretty challenging. But we're, we've been success.
[00:11:37] Matt Yonkovit: Now you talked about multi-tenancy. For a lot of people, multi-tenancy, um, if they've got one app, they might not think about it. Or even if they have multiple apps that need data, they're not necessarily thinking about the segregation. You mentioned cost savings there as a potential reason.
Maybe talk a little bit about that multi-tenancy and why that was attractive.
[00:11:56] Frank Kelly: Yeah. I mean for, we have a number of different customer [00:12:00] bases out there. Some are are larger, some are smaller. uh, yeah. So for some of them, again, with any environment, we're setting up databases. We're setting up Pulsar with its zookeepers and its bookies, and we're always saying, Well, for high availability.
We want, you know, at least two broker pods and we need a quorum in Booky. So I need three Booky pods. I need, all of a sudden I've got nine pods, I've got three zookeepers, three bookies, two bro brokers, and two proxy. So maybe that's 10. Right? So I've got 10 pods right off the the bat, and I need them be pretty snappy and responsive.
So giving them some CPU shares this memory. And if that's for a customer base where we wanna get like 50 or 60 customers online, that's even, that's pretty, you know, expensive to run. So we're trying to do is basically put clients together. They're logically separated, logical that the exact term logical tenant safety, if you will, in that environment.
So they're in separate, in the same Kubernetes name space, but different Pulsar, our name spaces with other controls. And, [00:13:00] uh, we can use that to save money basically so that those 10 pods can work for like five customers rather than just one.
[00:13:05] Matt Yonkovit: Yeah, so operationally it's more efficient because there's less to manage, but also, you know, you've got the, ability to use less, hardware and less components. So. Okay. That totally makes sense. You know, part of that is also the security of it, making sure that the data is segregated, which is nice to have that capability built in.
Cause that's an absolute requirement, um, as well. Yeah.
[00:13:29] Frank Kelly: So security was always key. So like, you know, all our ABS drives are encrypted. We're always making sure that, you know, we have proper controls, authorization, authentication within our spring applications and elsewhere. Um, we were, one of the good things we did, we use Isto is our service me, and we're able to bolt that on with Pulsar to make sure that, you know, there's some pretty strong controls around it.
I. PSR has his own authentication authorization mechanism, but we just didn't fancy spending a lot of time injecting certs and doing mtls when Isto will kind of do that for us, and it's been [00:14:00] pretty success.
[00:14:01] Matt Yonkovit: Oh, cool. Yeah. So you are, you have a team of developers who are using this and they're building on it and they're enhancing it. Um, maybe tell me a little bit about, like from your role, when you have developers who are coming on, what are the things that they find. Either interesting or odd about Pulsar.
Are there those things that you have to kind of explain the topics or explain, you know, what's going on?
[00:14:22] Frank Kelly: Subscriptions. That's what I find is the first thing. So I remember my first. Playing with pulse hour. All right. I created a topic, I wrote a message. I created a consumer. I listened to the topic end, nothing. What? I'm like, I'm used to cues. I write something to a queue, it stays there. I read from the queue, its, it's mine.
I'm like, What? I'm like, Spend hours. I'm like, What? So that's the big difference, right? Is that if the subscription wasn't preexisting, then. You're not gonna get the message cuz they're default for Java, at least I think for is to get latest message. So the minute you start, if I send a message at Time zero, I try to consume a time [00:15:00] one, it says give all message from time one forward.
And so you basically have to tell it all right, you gotta go back to earliest and get all the messages. And that's when I start learning about subscriptions. And then, you know, there's these things called backlogs, which sound like they're a queue backlog. There sound like there were a bunch of stuff in a queue, but they're not.
They're just like a logical marker. So that, that's pretty confusing to people, and that that part is the first part to get over is, is, um, what does the backlog mean? What does subscriptions mean? Um, you know, from a operational standpoint, the subscriptions are a little bit challenging too because other people can create subscriptions that are durable, they last forever, and if they last forever, you will never delete that data, right?
That, that old subscriptions hangs around Then. That, you know, pollster says, Well, if they haven't caught up, I'm just gonna hold onto that data just in case they come on, you know, on Tuesday, decide to catch up. But it's like it's a problem for retention. Cause all of a sudden I'm paying, well, [00:16:00] pretty dollar for all this EBS drives.
I'm like, where are all my disc space going? It's like, okay, we gotta got rogue subscription out there and gonna deleted.
[00:16:08] Matt Yonkovit: Hmm. I think anytime you change how you've thought about building an application or how a piece of technology works, it's often difficult to like make the transition because you just fall into old habits.
[00:16:18] Frank Kelly: That's right. Totally. And so you start thinking it's more, more of a streaming world of, you know, there's multiple people at different points in the queue, if you will, in the topic. Then it becomes a little bit easier to understand than.
[00:16:29] Matt Yonkovit: We were chatting, in a pre-call about, one of those other things, which is like the negative acknowledgement and how, developers are like, What about my exception handling?
[00:16:39] Frank Kelly: Right. Yeah. I mean, exceptionally is is always hard. I used to do this with MQ series back in the day, and it. I got a message, I can't handle it. I can't write the database. Oh, what do I do? And that's what we were doing is like write into a dead letter queue that somebody else pick it up. I'm like, Okay, now I've just created another software development problem and I've gotta reach for another queue and figure out how to handle it.
And what if somebody else already picked it up? And so the nice thing is [00:17:00] PTR gives this knack ability, which is if I can't handle just, you can either not acknowledge it, then a timer goes. In which case the message will be pre delivered or I can negatively acknowledge that I can handle this right now, somebody else can go handle it.
And so you can kind of basically say there's some errors that you can retry on, some errors, you can't. And so the negative acknowledgement lets you handle that, which is kind of nice because there there's times where we're trying to basically, um, you know, maybe offload data to s3 and, uh, you know, something happened For whatever reason, we couldn't, you know, the database was slammed and we couldn't get where the location was.
Retried 15 times, which we have configured. It's like, you know, every 10 seconds. Then, you know, if it doesn't work well, you know, then we have an alert. But, having to automatic the retry is automatic. Give you kind of a lot of flexibility.
[00:17:50] Matt Yonkovit: Oh, that's cool. Yeah. One of the other things that, you know, I've been reading a lot about is, the high availability, set up and the geographic replication available in Pulser [00:18:00] are, are, you know, I'm assuming that there is some, you know, geo specific things that you're doing.
and you are potentially taking use of that. So maybe tell us a little bit, about what that high availability looks like for, for you all and how you achieve that, especially at the throughput that you have. Um, you know, that, that's gotta be a, um, an interesting recovery time.
[00:18:22] Frank Kelly: Yeah, that's a great question. We really tend to use geographic replication. So we we're often, we just deploy really in the AWS region that's closest to our, to the customer. Uh, so it could be US East, S one, that kind of stuff. Um, but in terms of ha, I mean, we're always trying to be aware of, you know, Zuki has its own ha, right?
So it's gotta be three pods, but then the question is how are they spread out across nodes in Kubernetes? That across availability zones. So the, you know, the P star chart, the P star home chart has a bunch of anti affinity stuff, so it's certainly spreading things out across nodes. Um, and then, you know, one of things we have to still kind of go back and work on is, [00:19:00] um, is that also spread out across availability zones?
And my guess is it's not, and uh, you know, we, we've definitely the, that someone to work on for zookeeper and certainly on the bookies as well. Uh, it's even probably more critical, but the fact. If the zookeeper fails and bookings don't, it might as well be the same thing because we found that, you know, once those two things get outta sync, we're kind of, we're kind of hose, um, brokers are a little, they're stateful with a little more recoverable.
Um, so yeah, I think the HA is really having multiple pots spread across node. But the area to work on is, is the, um, the anti affinity. So breaking it out by availability zones.
[00:19:40] Matt Yonkovit: Okay. Well, and so you've been working with Pulsar for a while now. What's your favorite feature?
[00:19:46] Frank Kelly: What's my favorite feature? Um,
[00:19:51] Matt Yonkovit: Oh, there.
[00:19:52] Frank Kelly: I was, yeah, there's so many good things. So
just last night I was, , did [00:20:00] a production change and I just turned it on and I listened to it, to the, the audio streams, this log as a user and got listened to phone, phone calls going on. It's just seamless, right? So we're basically listening to the audio stream chunky in a, in the separate messages.
You know, it's kinda like Willy Wonka, that Mike TV moment. Do you remember that one? You like rips the kid apart, put some, opens it, and we're pull, pulling it all back together again. But it's, it's seamless, right? And it's just listening to it and it's like, I can't believe that's eight messages a second.
And it's a seamless, There's no, there's no glitches, There's no, um, you know,
[00:20:36] Matt Yonkovit: it all works together then really is
[00:20:37] Frank Kelly: just works. Yeah. And so that's working with compression and authentication and all behind the scenes going across, you know, uh, multiple VPCs and, you know, um, to into memory and onto disk and then back out. And then it's kinda like, it's pretty cool.
It's pretty cool. It just works. That part just works. The compression is awesome too. I mean, that, we just turned that on and we got like a three x [00:21:00] improvement in, in a lot of our performance benchmarks and IO and, and,
[00:21:04] Matt Yonkovit: Oh really? Wow. That's, That's a pretty big savings.
[00:21:07] Frank Kelly: I guess web isn't as, isn't as, uh, efficient as one would things. There's certainly other, um, audio formats that we're looking at that would be, you know, compressed by default.
But it was kinda nice to get that out the box. Cause we were looking at staring down 500 megabytes per second. Then all of a sudden we just went to 150. It was like, yeah. Sweet. That's much
[00:21:25] Matt Yonkovit: Well, yeah, that, that, that does, that does reduce the costs and
[00:21:27] Frank Kelly: It does.
[00:21:28] Matt Yonkovit: the bandwidth there. Um, so as people are deploying Pulser and, and as your users or developers are starting to use it, Is there a common like problem that continually comes up, like, you know, an issue that you see over and over again?
[00:21:42] Frank Kelly: Common problem.
I don't chose a common problem, but there's a common piece of advice, which is Pulser, uh, exposes a whole bunch of metrics for Prometheus and that, um, you really need to graph those.
[00:21:59] Matt Yonkovit: [00:22:00] Ah.
[00:22:00] Frank Kelly: for the proxy, the broker, the bookies, the zookeeper. That will be your lifesaver. I mean, for us, we're kind of a somewhat of a unique use case in the sense that we're doing all these binary streams and stuff like that.
So for us, we're kind of really monitoring broker, uh, ledger, CAEs and um, you know, that kind of stuff. But in terms of, you know, stiff goes wrong, uh, we had an issue this morning, right? You know, we, all of a sudden our booky response times went through really high. And so we perme these alerting kind of goes.
You're Booky alert, you're booky response time. P 99. Just hit 30 seconds. I'm like, Okay, . That's not good. That's not good. What the heck? So we looked at it and was like, it was a big read load. Where'd that reload come from? Right? And then having those tools to track that down, cuz between the proxies, the brokers, the bookies, the zookeepers, your, your drives, all the pods, debugging is gonna be hard.
And so, you know, upfront, invest that time. You know, whether it's your pana charts, uh, alerting, [00:23:00] know what goes wrong and alert on it and be ready. Uh, cuz that's, that's the hard part you, because, and we also wrote run books, right? It's like, hey, if you see this, Lord, look here cuz you're gonna get it at three o'clock in the morning and you're kinda gonna go, uh, I'm not caffeinated.
What do I do? Like, okay. I wrote myself a room book. Okay, Dear Frank, here's what you're supposed to do. Like, thanks, so,
[00:23:21] Matt Yonkovit: Frank, I know you were, you were gonna speak at the Pulsar Summit, but you couldn't make it this
[00:23:24] Frank Kelly: I know that
[00:23:25] Matt Yonkovit: was the topic you were gonna give?
[00:23:27] Frank Kelly: So it was basically, I gave a topic the prior year and uh, it was more of the, more of a things we discovered. So first year was all about, we discovered to, um, so this is another little thing. They always say, Well, you know, we, we write, we write to the journal, but you read from the ledger, right?
And it's like, and the two never really meet and like, well, they have to, Right? The journal has to. Written to the ledger. And so we learned that if you're reading for the ledger at the same time as you're trying to write to the journal, write as the journal's trying to write to the ledger you're trying to read for the ledger, you can basically jam up the bookies.
[00:24:00] And so what we learned was, again, we're in this for, for unfortunate real time case. If we've made the broker cash is really big, then we would just read for the broker cash and never hit the bookies. And then all of a sudden our throughput was off the races. So that was one big positive from 20 21, 20 22 was when I realized that.
We would be creating a lot of topics, um, you know, very, very quickly. And, uh, for that we, you know, we needed to improve zookeeper and a lot of response times there around zookeeper and scaling zookeeper. And so zookeeper has a number of settings, um, so of which are just, you know, now available to improve that scalability.
So what that metadata, zookeepers, and metadata store keeps track of all the topics. And as they're created and deleted, it gets quite a bit of. And so that was a key there to, to, to monitor that.
[00:24:49] Matt Yonkovit: Yeah, so I mean, it's just really about the scale. Now you mentioned like you're creating a lot of topics because you create a topic per phone.
[00:24:56] Frank Kelly: Yeah. So for every phone call, those two topics, there's one which is the [00:25:00] audio and one which is the, um, the, the, the analytics. And we had multiple debates about this, about what the way to go. And we wanted to do maybe shared subscriptions, a big fat topics. And that was kind of suggestion. But what we realized with the, the kind big fat topics, Our, our AI algorithms are kind of, you know, small little pods, thousands of them, thousands and thousands of pods, and they were just like, I want one audio stream and I'll produce one, one, uh, analytics stream.
Great. The problem is if we have one big fat pipe, um, that would, that fat pipe of IO would have to go all to the, um, that single pod and it would only have to pick like one 1000 to one 5000th of that io. It would be interested. . And so, um, I know those talking future for doing some, uh, filtering basically, uh, on the server side, but we felt it was just easier to manage and get go live.
We had, you know, one topic per audio stream and then, you know, um, it worked out pretty well. But there, [00:26:00] it's not the default set up for Pulser to think about that. So there was a number. Settings we had to tweak. I can find my, my chalk from 2021 about that. Um, but you know, as, as with everything in Pollstar, there's a setting for it.
You just have to find out what it is.
[00:26:16] Matt Yonkovit: Yeah. No, and it's, it's interesting because. You know, it, it goes to show you, you know, as you're building out applications, uh, especially as you grow and scale, oftentimes you need to think about the workload and optimize, you know, the infrastructure for the workload, not just on the setting side, but in your, your, your case.
You're like, Oh, well one topic, per phone call makes a lot of sense for us. That's not something. You would necessarily think about, but I think that as we start to think about like the future of, you know, how things grow, how things evolve, those are the types of design decisions that a lot of people might not realize they need to make early on or as early as possible.
[00:26:59] Frank Kelly: [00:27:00] yeah, yeah. I was just thinking today, just like, you know, if there's a lesson I'd learned in 20 plus years, it was like, don't front your database with a queue, cuz you, you can jam messages into your queue faster than you can jam them into the database, those kind of things. But you really have to be aware of rough order of magnitude of, of each thing.
Right? So databases you're talking about, say a 10 millisecond insert. Maybe now it's locally or maybe it's not, you know, me cash might, you know, it's one millisecond, whatever, but that'd be different. But then you gotta understand the order of magnitude of things and we can jam tons of messages into P Star.
We're like looking at, we have record everything. We're like, Oh, P star send times, like less than a millisecond. I'm like, Yeah, you can really, you know, overwhelm things downstream if you want to with, uh, with Pulsar as the front end. So you really be aware of proper choices of storage, um, and not just for.
Scale, but also cost, right? So we were trying to jam stuff into, um, our bookies, and then we realized, Oh, let's just pick [00:28:00] GP three drives. They're really, really fast and you know, they're inexpensive compared to iops, optimized the I two s and, uh, you know, was expensive. We still paying a lot of money, and we realized, well, if we just add more bookies and had GP two achieve the same scale with uh, less cost.
And, but yeah, you have to really be thinking. The relative order of magnitude of those, those bits. Um, again, if you're just sending a couple messages around, you know, every couple of seconds, you're not too worried about it. But if you're really at the high scale world, um, yeah, you're gonna be living and breathing
[00:28:31] Matt Yonkovit: There's a fine line between over or early optimization or future proofing, like spending too much time there and, and not spending enough time. Uh, and that's a whole interesting, you know, debate and discussion, but
[00:28:42] Frank Kelly: I think layering, make sure you have the proper interfaces so you can swap out implementations is always important. That's one of the things we were also gonna do as well, is we have our, our telephony systems and our compute systems are directly tied into Pulsar with the library that we wrote. And now we're like, You know what?
We wanna pull that out and abstract that away just because. , [00:29:00] um, you know, we're often like, Oh, we gotta change the library and add this little setting. NA's gotta go to our, you know, telephony team. And then they have to deploy it and they've gotta make sure that's deployed everywhere. And that's the thing we have to organize.
It's like, well, we had an interface in front that was abstracted away then, you know, that, that would be a lot easier. So,
[00:29:18] Matt Yonkovit: Oh, very
[00:29:19] Frank Kelly: yeah. Well, a lot of lessons learned. Good.
[00:29:21] Matt Yonkovit: Ah, well, Frank, I wanna thank you for, spending some time with me and, telling me about how you're using Pulsar. Um, helping me learn a little bit more about, how people are deploying it and using it, and some of those things we gotta watch out for. I really do appreciate it.
[00:29:36] Frank Kelly: Not all. Thank you very much. It was an absolute pleasure.
