speaker-0: We've moved to this new realm of machine learning where the agent loop becomes a new machine learning modality where continuous feedback from these models that feeds back into the model on real world data, right? Like on real world scenarios, that becomes its own essentially machine learning technique.
speaker-1: Welcome to the incremental marketer, a content series where we meet with top marketing executives and leaders across industries to help you drive growth in your business. Welcome everyone to the incremental marketer. Today's guest is Piotr Motikewicz. Very excited to chat. He leads AI engineering and operations at SlingWave. Piotr is a technologist who's been deep in the code and the business for over two decades. He's built and scaled high traffic platforms from early social networks serving millions of users to leading complex cloud migrations and product transformations. Before SlingWave, Pio headed up the cost anomaly detection team at Amazon Web Services, AWS. He previously served as CTO of Interactive One as well, where he led platforms reaching over a million monthly users and hundreds of millions of page views. He's driven full stack rebuilds, database migrations, mobile launches, and large scale moves to AWS. And at SlingWave, Pio sits at the intersection of engineering rigor and marketing reality. helping take the platform to the next level with new capabilities across analytics, experimentation, and AI-driven insights. Today, we're digging into how great engineering enables better marketing decisions, and we're unpacking what we've just launched at SlingWave to truly stand apart from the competition. Pia, it's great to have you on the program.
speaker-0: Yeah, first and foremost, Paul, thank you for having me. Yeah, I'm excited to dive into these topics. I think there's a lot to cover here.
speaker-1: Love it. We do have a lot and you you and I started working together over a year ago now. It's been a, a fast ramp into the world of AI driven products and engineering development. So I'd love it if you could tell us about your career. So your, your origin story. I know you've built your career building and scaling complex systems. Tell us about your journey.
speaker-0: Yeah, it's been a year, but like it feels like a decade, right? Because the AI space is moving so fast. But yeah, before that, you know, I sort of started out in social networking space and in New York, you know, believe it or not, New York city was sort of hotspot for early social networking platforms back in the day. that, you know, that's where I sort of cut my teeth as a backend engineer and sort of. You know, got to really learn how to scale systems back then. It was challenging because we didn't have a lot of, we didn't have cloud. So, so, so this was where I sort of picked up my engineering skills there. And then, you know, recently I spent a couple of years at AWS where I led up a team that built anomaly detection for, for AWS customers, ⁓ to, ⁓ monitor their costs. And this was interesting, ⁓ because you know, the, range of ⁓ accounts that the range of data sets that we dealt with there all the way from the smallest accounts to the largest enterprises ⁓ ever that use AWS. The data volume was quite large. And so we did tap into quite a bit of machine learning techniques to be able to do that. that actually is paying dividends as we're working in AI these days.
speaker-1: ⁓ so broad range from where you started and then, and AWS, how did you see the engineering function change across that, that time period?
speaker-0: Yeah. So, honestly, like in the initial stage, right? Like even from the 2000s, even as we moved to cloud, the sort of your tools were changing, but now your function, right? As for example, a CTO, a lead of engineering. mean, these days in the last year, especially, I think we're seeing actually significant changes in how those roles ⁓ operate and essentially what it means, for example, to be a CTO, right? what it means to be a software engineer. So that's actually changing rapidly, especially in the last year. So even though I did not see a lot of change in 2000s, even as we were moving from on-prem to cloud, I feel like these days there's quite a bit of change in that. So that adds to that sort of complexity and dynamic that is occurring in this space.
speaker-1: want to drive into the culture of what you went through it at both interactive one and then AWS, because you've taught our teams and many teams that you advise some processes and some cultural really guiding principles in terms of how to develop. I'm curious, can you just talk through those and how are they different now if they are.
speaker-0: Up until a year ago, know, sure, AWS operated at a different scale, but ultimately you had a clear sort of swim lanes, right? Clear progression from a junior engineer to a senior engineer to a manager versus now there's this term of token budget, right? Like, and that almost determines your, as an engineer, right? Like your ability to move at a certain pace. These days, as an engineer, you're almost becoming a manager, right? So that transformation, I've seen that at ⁓ SlingWave as well, takes time to adopt, but it is really sort of a mind shift, like a ⁓ paradigm shift, how you operate as an engineer is that you almost are becoming a manager. So those skills, managerial skills are actually becoming more important and necessary for even a staff engineer.
speaker-1: Yeah. And token budget as a great example, what's a real world use case of that and how you've seen it impact or just kind of direct an organization in terms of how they're building.
speaker-0: Yeah. So, so back in the day, you know, we had this metric of like, okay, how many lines of code you are producing as an engineer. And today that's becoming less and less important because, you know, producing code is easier and easier. Sort of you're moving from essentially, ⁓ writing lines of code to offloading your cognitive ability to these agents, agentic workflows. And the more of that you do, and obviously, you know, you supervise all of that, the more sort of tokens that agent or those agents consume, right? So you start measuring essentially your output in terms of as an engineer, right? As a product engineer, as an FPM, you start measuring that in, okay, how many tokens your agents consumed, right? And so that's becoming one of the measures and a way to assess whether effectively utilizing the technology.
speaker-1: Now, one of the things that's interesting, I know about your background and your, you personally is you have this blend of products and business mindset plus engineering CTO leader. And if you've worked with leaders like Tom Newman at, interactive one and within Kelly Newman portfolio, KNA very closely aligned with the business. What do you think your perspective or how does that perspective really change the way that you build product?
speaker-0: This is sort of always has been my MO, right? So from the beginning, you know, when I sort of began my journey, I was always sort of builder first. And obviously the technique, the craft is important and you pick that up as you go and you develop your style. But really the output, the business impact was what I always cared about. And I feel like it's a shift that's occurring right now that This type of sort of approach is becoming even more important because, you know, matters less again, how many lines of code you produce and more what it does. What is the output? What is the, what is the business value? So that's, that's, that's what I felt like my sweet spot of like how I operate always lied as sort of an intersection of, you know, the business and tech and product, right? I would say. I'm leaning more into that even more, right? So as I help these businesses scale, really the business value is where you want to focus on first and how you get there. That continues to change. You know, our workflows continue to change and what works six months ago, maybe doesn't work anymore, but there's a better, newer way of approaching the problem. So that continues to evolve. Staying current is more important than ever. So as an engineer, as a product builder, staying current on the latest stack, on the latest set of tools that you can use is becoming even more important because that gap is widening. ⁓
speaker-1: Let's talk about that a bit because you've worked at large organizations, which are maybe not as agile as the startups and the entrepreneurial endeavors you've undertaken. So for our listeners, how do you get that culture and the mechanics into your organization of using the latest and greatest, no matter what size you are? Does it matter?
speaker-0: Yes, it matters. So here's one heuristic that you can think through as you're thinking about these new workloads, these new ways of using these systems is that it generally works better when you're starting from fresh start versus you're trying to work with an existing system, a legacy system that's been around, that there's a lot of context of internal tribal knowledge that's A lot of times just in the heads of the people that built it. So that's tougher, right? To apply the latest sort of way of working, which, know, by the way, as we rebuilt SlingWave, that was the opportunity, right? Because we were able to sort of start from scratch with all the knowledge that we had from the past, be able to build something new here that allowed us to move faster because it is sort of a natively built from scratch, you know, product with the with the, with the latest. So, so that's what I would say is like the biggest difference is it is tougher to apply these new techniques to an existing older code bases.
speaker-1: Is it easier to just rebuild from scratch? Like you said, so refactor and optimize what you have.
speaker-0: And that's a judgment call and that call also changes as something that maybe wasn't possible, you know, three months ago is possible today with the latest model, right? Like, that answer to that question continues to change and you have to have that experience, right? Of having built these larger systems in the old way and then also experience with the latest and greatest techniques, right? And then be able to sort of make that judgment call at that time, like... Does it make sense to start from scratch for this particular project or not? So yeah, that's a great question and continues to be an important topic.
speaker-1: I know you've been doing a lot with claw bot and open claw and the team has worked in cursor and in cloud code and it's changing so quickly. I want to just start with the problem you're solving with the team here at sling wave. So we have relaunched the platform. Like you said, we had a big announcement at CES. It's amazing how quickly the platform has been able to start to be impactful for our customers on, you know, the framework of The tech, the ML that we've built over multiple years with our MMM. So from an engineering standpoint, you know, what is the problem that we're trying to solve? And starting with every platform we work on grades, its own homework. Why do think the industry hasn't cracked this yet?
speaker-0: I think our product is uniquely positioned in a way where we're not selling media. So there's no incentive on our side to over credit. So I think that plays into our ability to build a system that's trustworthy. We see the customer journey, like there's multiple platforms. So there's some crossover between them that we can distill and learn from. we've got these ways of not just modeling ⁓ what could have happened if we spend the money, but also what could have happened if we didn't spend the money, right? So because we built this simulator, right? We're able to ⁓ look at the analysis from many different angles. And so, yeah, so it's... high level, it's just that there is no incentive, right? Like it's about aligning incentives and the incentives are set up the right way, which is in this case, I believe they are, right? Because we're not selling media, we're simply building a system as transparently as possible. We're trying to determine what's the best outcome for the customer.
speaker-1: That's it's a great point. think the gap of platforms grade their own homework and they do want advertisers to have the best experience, but it's competitive out there and they're, they're in a silo by nature. They can't see what's happening on the other platforms. So they're at a disadvantage. And to your point that we have the ability to model with a predictive model and with our AI layer, factual and counterfactual data points that allow us to understand those, those dynamics. So. When you think about the architecture that we've built, have have MMM and we have attribution, have experimentation and the AI layer in one system. Technically speaking, what makes that actually unified versus stitched together processes that are not talking to each other.
speaker-0: Yeah, yeah, that's a great question. So I think really the biggest breakthrough, right? That we've been able to build and you know, it wasn't obvious, right? Like nine months ago, this was, I think, not obvious, right? We were making these calls that yes, let's go in that direction, but is it going to work out? You know, we weren't sure. And that's part of the challenge. Like as we're operating in this environment, you sort of have to, you know, as the old saying goes, you know, skate to where the puck is going, but You know, you do have to sort of predict, right, where the puck is going to go and that's not always obvious. So in this case, what we really double down on is the fact that, you know, these leading models are going to continue to improve and the sort of the small signs of what we were seeing nine months ago, 12 months ago are directionally, that's where things are going to go. really what the solution is, is feeding the LLMs the right context. from all these disparate data sources. So to your point, how do we do this? We take the MMM output, we take the attribution outputs, we take all kinds of data sources, use ML and LLMs to reconcile, to normalize the data, and then ask the LLMs for the output, right? And then there's a lot that goes into iterative process of figuring out what works, what doesn't, measuring the results. evaluating the results. My high level point is that this is now working, right? But it wasn't obvious 12 months ago that this is actually going to work.
speaker-1: There's a lot of naysayers and or edge cases or fail cases as we try it.
speaker-0: Yes, and different also different ways of using this system, right? So just because the system outputs a certain prediction and I want to stress the point that today these systems are producing discrete outputs, right? That are reliable, right? Again, like six months ago that wasn't the case. So now we have these discrete outputs, right? For example, for a budget reallocation, right? Like we have a discrete output that comes out with numbers, right, for a specific reallocation. So that's great. It checks out, the math checks out. But this isn't, you know, a zero one switch where you all of the sudden have to go and apply that, right? Like you can, for example, use this output, compare this to maybe what a human could do or the previous allocation, and then come up with a narrative. And now it's not necessarily about closing the loop right away, but for the human operator, the analyst, having a narrative around what we should do and why. Because then there's a discussion with CMO or somebody else, like, is the new reallocation, but also why we should be doing it. so that's where the usefulness comes in, right? So it's not necessarily about just, you blindly reallocating budget, but having that introspection. And sometimes it's just the starting point for the discussion and sometimes it's ready to go. But that's the usefulness of these systems that I think wasn't also very apparent.
speaker-1: Right. We have a, an evolution that is accelerating here and the architecture that we've chosen embraces that acceleration. So I want to ask you that question about being open to the fact that the models are changing and improving so quickly. What we thought was not possible is now possible. Now what we still think is not possible is going to be possible. What do you think is the promise of incrementality in days? Not months or years like this process, this MMM process used to be a six month or a 12 month engagement, large teams, highly custom consulting services. And that layer still exists, but what has needed to change architecturally to make that promise of incrementality and days, not months.
speaker-0: Why is this possible now and what are we doing? A year ago, you had the chat bots, even four years before that, before the latest family of AI models came out. But those were a thermal in a way. You didn't have a lot of context in them from previous discussions. it was useful, but sometimes it would hallucinate. You really didn't have a feedback loop. And what we found out on top of that, beyond the feedback loop, is really ⁓ helping the LLMs. The LLMs are, you can think of them as these sort of like, these brains that are in this vacuum, right? And so that's how I like to sort of put myself in the position of an LLM, right? Like it's as if I was trying to answer a question and This is the context, this is the feedback that I'm getting, but I don't know any of the context. I don't know what these terms mean. I don't know what this business is about. And so what we found is when we employed sort of this idea of ontology and the idea that there's layers to this and there's maybe the broad industry layer that defines certain aspects. even certain macro environments today, the tariffs are higher than they were a month ago, and then maybe next week they're gonna change again. So all this information goes into these ontology layers. And we found that when we actually structure the input, the context that goes into the LLM and really give the LLM a definition of the terms, right, that we're asking it to solve for. You know, in marketing, are specific terms, ROAS, spend cars, and all these terms that experienced marketer or analysis know instinctively and intuitively. But if you don't define those things for the LLMs, you know, some of it the LLMs already know because they do have that broad sort of world knowledge because they scrape all of the internet. But there are terms that are specific to the company, right? If a client that uses our platform sells digital goods in a specific vertical, right? Like there are certain aspects of that that are specific to that business. Just giving the LLMs a lot more context and a lot more structure context and that feedback loop. The other big important piece of this is as we were describing earlier, that reallocation, seeing what happens with that reallocation and then feeding it back to the LLM as well as actual human feedback. We're actually telling the model what went well, what didn't go well, right? With that reallocation. So building that system and feeding it back into the model and a couple of other ways. Seeing an improvement where you went from something that just wasn't working to something that's actually producing usable results that have real impact on ⁓ marketing spend and effectiveness.
speaker-1: Love the examples and all the way from ontology to how do we go faster with multiple LLMs, which we can dig into on a future episode excited about the progress the team's made. But I wanted to ask you as we close is AI in this current era of marketing, it's all the buzz, obviously. What do you, what excites you and what are you seeing as you look over the hedge at what's, what's ahead of us?
speaker-0: I feel like agentic and agents, these agentic loops that were going in that direction, but they weren't really fully there. Now, I almost think of the agentic workflows and these capabilities as sort of like a new modality of machine learning. So as we were pre-training and then post-training and these big labs that have been doing that and continue to do that and continue to see improvements, we've moved... as well into this new sort of realm of machine learning where the agent loop becomes a new machine learning modality where that continuous feedback. And then this is, think where we've zeroed in at SlingWave is this continuous feedback from these models that feeds back into the model on real world data, right? Like on real world scenarios. That becomes its own essentially machine learning technique. This is something that I'm actually excited about. And I think the next six to 12 months are going to see step change improvement.
speaker-1: Pio, this has been great. Thank you. And for all our listeners out there, thank you for joining us on today's episode of the incremental marketer. Stay tuned for more voices, more lessons, and more proof that incremental moves can drive big growth. If you enjoyed this conversation, don't forget to subscribe, like, and share, and join us for more episodes with the leaders shaping the future of marketing.
speaker-1: Welcome to the incremental marketer, a content series where we meet with top marketing executives and leaders across industries to help you drive growth in your business. Welcome everyone to the incremental marketer. Today's guest is Piotr Motikewicz. Very excited to chat. He leads AI engineering and operations at SlingWave. Piotr is a technologist who's been deep in the code and the business for over two decades. He's built and scaled high traffic platforms from early social networks serving millions of users to leading complex cloud migrations and product transformations. Before SlingWave, Pio headed up the cost anomaly detection team at Amazon Web Services, AWS. He previously served as CTO of Interactive One as well, where he led platforms reaching over a million monthly users and hundreds of millions of page views. He's driven full stack rebuilds, database migrations, mobile launches, and large scale moves to AWS. And at SlingWave, Pio sits at the intersection of engineering rigor and marketing reality. helping take the platform to the next level with new capabilities across analytics, experimentation, and AI-driven insights. Today, we're digging into how great engineering enables better marketing decisions, and we're unpacking what we've just launched at SlingWave to truly stand apart from the competition. Pia, it's great to have you on the program.
speaker-0: Yeah, first and foremost, Paul, thank you for having me. Yeah, I'm excited to dive into these topics. I think there's a lot to cover here.
speaker-1: Love it. We do have a lot and you you and I started working together over a year ago now. It's been a, a fast ramp into the world of AI driven products and engineering development. So I'd love it if you could tell us about your career. So your, your origin story. I know you've built your career building and scaling complex systems. Tell us about your journey.
speaker-0: Yeah, it's been a year, but like it feels like a decade, right? Because the AI space is moving so fast. But yeah, before that, you know, I sort of started out in social networking space and in New York, you know, believe it or not, New York city was sort of hotspot for early social networking platforms back in the day. that, you know, that's where I sort of cut my teeth as a backend engineer and sort of. You know, got to really learn how to scale systems back then. It was challenging because we didn't have a lot of, we didn't have cloud. So, so, so this was where I sort of picked up my engineering skills there. And then, you know, recently I spent a couple of years at AWS where I led up a team that built anomaly detection for, for AWS customers, ⁓ to, ⁓ monitor their costs. And this was interesting, ⁓ because you know, the, range of ⁓ accounts that the range of data sets that we dealt with there all the way from the smallest accounts to the largest enterprises ⁓ ever that use AWS. The data volume was quite large. And so we did tap into quite a bit of machine learning techniques to be able to do that. that actually is paying dividends as we're working in AI these days.
speaker-1: ⁓ so broad range from where you started and then, and AWS, how did you see the engineering function change across that, that time period?
speaker-0: Yeah. So, honestly, like in the initial stage, right? Like even from the 2000s, even as we moved to cloud, the sort of your tools were changing, but now your function, right? As for example, a CTO, a lead of engineering. mean, these days in the last year, especially, I think we're seeing actually significant changes in how those roles ⁓ operate and essentially what it means, for example, to be a CTO, right? what it means to be a software engineer. So that's actually changing rapidly, especially in the last year. So even though I did not see a lot of change in 2000s, even as we were moving from on-prem to cloud, I feel like these days there's quite a bit of change in that. So that adds to that sort of complexity and dynamic that is occurring in this space.
speaker-1: want to drive into the culture of what you went through it at both interactive one and then AWS, because you've taught our teams and many teams that you advise some processes and some cultural really guiding principles in terms of how to develop. I'm curious, can you just talk through those and how are they different now if they are.
speaker-0: Up until a year ago, know, sure, AWS operated at a different scale, but ultimately you had a clear sort of swim lanes, right? Clear progression from a junior engineer to a senior engineer to a manager versus now there's this term of token budget, right? Like, and that almost determines your, as an engineer, right? Like your ability to move at a certain pace. These days, as an engineer, you're almost becoming a manager, right? So that transformation, I've seen that at ⁓ SlingWave as well, takes time to adopt, but it is really sort of a mind shift, like a ⁓ paradigm shift, how you operate as an engineer is that you almost are becoming a manager. So those skills, managerial skills are actually becoming more important and necessary for even a staff engineer.
speaker-1: Yeah. And token budget as a great example, what's a real world use case of that and how you've seen it impact or just kind of direct an organization in terms of how they're building.
speaker-0: Yeah. So, so back in the day, you know, we had this metric of like, okay, how many lines of code you are producing as an engineer. And today that's becoming less and less important because, you know, producing code is easier and easier. Sort of you're moving from essentially, ⁓ writing lines of code to offloading your cognitive ability to these agents, agentic workflows. And the more of that you do, and obviously, you know, you supervise all of that, the more sort of tokens that agent or those agents consume, right? So you start measuring essentially your output in terms of as an engineer, right? As a product engineer, as an FPM, you start measuring that in, okay, how many tokens your agents consumed, right? And so that's becoming one of the measures and a way to assess whether effectively utilizing the technology.
speaker-1: Now, one of the things that's interesting, I know about your background and your, you personally is you have this blend of products and business mindset plus engineering CTO leader. And if you've worked with leaders like Tom Newman at, interactive one and within Kelly Newman portfolio, KNA very closely aligned with the business. What do you think your perspective or how does that perspective really change the way that you build product?
speaker-0: This is sort of always has been my MO, right? So from the beginning, you know, when I sort of began my journey, I was always sort of builder first. And obviously the technique, the craft is important and you pick that up as you go and you develop your style. But really the output, the business impact was what I always cared about. And I feel like it's a shift that's occurring right now that This type of sort of approach is becoming even more important because, you know, matters less again, how many lines of code you produce and more what it does. What is the output? What is the, what is the business value? So that's, that's, that's what I felt like my sweet spot of like how I operate always lied as sort of an intersection of, you know, the business and tech and product, right? I would say. I'm leaning more into that even more, right? So as I help these businesses scale, really the business value is where you want to focus on first and how you get there. That continues to change. You know, our workflows continue to change and what works six months ago, maybe doesn't work anymore, but there's a better, newer way of approaching the problem. So that continues to evolve. Staying current is more important than ever. So as an engineer, as a product builder, staying current on the latest stack, on the latest set of tools that you can use is becoming even more important because that gap is widening. ⁓
speaker-1: Let's talk about that a bit because you've worked at large organizations, which are maybe not as agile as the startups and the entrepreneurial endeavors you've undertaken. So for our listeners, how do you get that culture and the mechanics into your organization of using the latest and greatest, no matter what size you are? Does it matter?
speaker-0: Yes, it matters. So here's one heuristic that you can think through as you're thinking about these new workloads, these new ways of using these systems is that it generally works better when you're starting from fresh start versus you're trying to work with an existing system, a legacy system that's been around, that there's a lot of context of internal tribal knowledge that's A lot of times just in the heads of the people that built it. So that's tougher, right? To apply the latest sort of way of working, which, know, by the way, as we rebuilt SlingWave, that was the opportunity, right? Because we were able to sort of start from scratch with all the knowledge that we had from the past, be able to build something new here that allowed us to move faster because it is sort of a natively built from scratch, you know, product with the with the, with the latest. So, so that's what I would say is like the biggest difference is it is tougher to apply these new techniques to an existing older code bases.
speaker-1: Is it easier to just rebuild from scratch? Like you said, so refactor and optimize what you have.
speaker-0: And that's a judgment call and that call also changes as something that maybe wasn't possible, you know, three months ago is possible today with the latest model, right? Like, that answer to that question continues to change and you have to have that experience, right? Of having built these larger systems in the old way and then also experience with the latest and greatest techniques, right? And then be able to sort of make that judgment call at that time, like... Does it make sense to start from scratch for this particular project or not? So yeah, that's a great question and continues to be an important topic.
speaker-1: I know you've been doing a lot with claw bot and open claw and the team has worked in cursor and in cloud code and it's changing so quickly. I want to just start with the problem you're solving with the team here at sling wave. So we have relaunched the platform. Like you said, we had a big announcement at CES. It's amazing how quickly the platform has been able to start to be impactful for our customers on, you know, the framework of The tech, the ML that we've built over multiple years with our MMM. So from an engineering standpoint, you know, what is the problem that we're trying to solve? And starting with every platform we work on grades, its own homework. Why do think the industry hasn't cracked this yet?
speaker-0: I think our product is uniquely positioned in a way where we're not selling media. So there's no incentive on our side to over credit. So I think that plays into our ability to build a system that's trustworthy. We see the customer journey, like there's multiple platforms. So there's some crossover between them that we can distill and learn from. we've got these ways of not just modeling ⁓ what could have happened if we spend the money, but also what could have happened if we didn't spend the money, right? So because we built this simulator, right? We're able to ⁓ look at the analysis from many different angles. And so, yeah, so it's... high level, it's just that there is no incentive, right? Like it's about aligning incentives and the incentives are set up the right way, which is in this case, I believe they are, right? Because we're not selling media, we're simply building a system as transparently as possible. We're trying to determine what's the best outcome for the customer.
speaker-1: That's it's a great point. think the gap of platforms grade their own homework and they do want advertisers to have the best experience, but it's competitive out there and they're, they're in a silo by nature. They can't see what's happening on the other platforms. So they're at a disadvantage. And to your point that we have the ability to model with a predictive model and with our AI layer, factual and counterfactual data points that allow us to understand those, those dynamics. So. When you think about the architecture that we've built, have have MMM and we have attribution, have experimentation and the AI layer in one system. Technically speaking, what makes that actually unified versus stitched together processes that are not talking to each other.
speaker-0: Yeah, yeah, that's a great question. So I think really the biggest breakthrough, right? That we've been able to build and you know, it wasn't obvious, right? Like nine months ago, this was, I think, not obvious, right? We were making these calls that yes, let's go in that direction, but is it going to work out? You know, we weren't sure. And that's part of the challenge. Like as we're operating in this environment, you sort of have to, you know, as the old saying goes, you know, skate to where the puck is going, but You know, you do have to sort of predict, right, where the puck is going to go and that's not always obvious. So in this case, what we really double down on is the fact that, you know, these leading models are going to continue to improve and the sort of the small signs of what we were seeing nine months ago, 12 months ago are directionally, that's where things are going to go. really what the solution is, is feeding the LLMs the right context. from all these disparate data sources. So to your point, how do we do this? We take the MMM output, we take the attribution outputs, we take all kinds of data sources, use ML and LLMs to reconcile, to normalize the data, and then ask the LLMs for the output, right? And then there's a lot that goes into iterative process of figuring out what works, what doesn't, measuring the results. evaluating the results. My high level point is that this is now working, right? But it wasn't obvious 12 months ago that this is actually going to work.
speaker-1: There's a lot of naysayers and or edge cases or fail cases as we try it.
speaker-0: Yes, and different also different ways of using this system, right? So just because the system outputs a certain prediction and I want to stress the point that today these systems are producing discrete outputs, right? That are reliable, right? Again, like six months ago that wasn't the case. So now we have these discrete outputs, right? For example, for a budget reallocation, right? Like we have a discrete output that comes out with numbers, right, for a specific reallocation. So that's great. It checks out, the math checks out. But this isn't, you know, a zero one switch where you all of the sudden have to go and apply that, right? Like you can, for example, use this output, compare this to maybe what a human could do or the previous allocation, and then come up with a narrative. And now it's not necessarily about closing the loop right away, but for the human operator, the analyst, having a narrative around what we should do and why. Because then there's a discussion with CMO or somebody else, like, is the new reallocation, but also why we should be doing it. so that's where the usefulness comes in, right? So it's not necessarily about just, you blindly reallocating budget, but having that introspection. And sometimes it's just the starting point for the discussion and sometimes it's ready to go. But that's the usefulness of these systems that I think wasn't also very apparent.
speaker-1: Right. We have a, an evolution that is accelerating here and the architecture that we've chosen embraces that acceleration. So I want to ask you that question about being open to the fact that the models are changing and improving so quickly. What we thought was not possible is now possible. Now what we still think is not possible is going to be possible. What do you think is the promise of incrementality in days? Not months or years like this process, this MMM process used to be a six month or a 12 month engagement, large teams, highly custom consulting services. And that layer still exists, but what has needed to change architecturally to make that promise of incrementality and days, not months.
speaker-0: Why is this possible now and what are we doing? A year ago, you had the chat bots, even four years before that, before the latest family of AI models came out. But those were a thermal in a way. You didn't have a lot of context in them from previous discussions. it was useful, but sometimes it would hallucinate. You really didn't have a feedback loop. And what we found out on top of that, beyond the feedback loop, is really ⁓ helping the LLMs. The LLMs are, you can think of them as these sort of like, these brains that are in this vacuum, right? And so that's how I like to sort of put myself in the position of an LLM, right? Like it's as if I was trying to answer a question and This is the context, this is the feedback that I'm getting, but I don't know any of the context. I don't know what these terms mean. I don't know what this business is about. And so what we found is when we employed sort of this idea of ontology and the idea that there's layers to this and there's maybe the broad industry layer that defines certain aspects. even certain macro environments today, the tariffs are higher than they were a month ago, and then maybe next week they're gonna change again. So all this information goes into these ontology layers. And we found that when we actually structure the input, the context that goes into the LLM and really give the LLM a definition of the terms, right, that we're asking it to solve for. You know, in marketing, are specific terms, ROAS, spend cars, and all these terms that experienced marketer or analysis know instinctively and intuitively. But if you don't define those things for the LLMs, you know, some of it the LLMs already know because they do have that broad sort of world knowledge because they scrape all of the internet. But there are terms that are specific to the company, right? If a client that uses our platform sells digital goods in a specific vertical, right? Like there are certain aspects of that that are specific to that business. Just giving the LLMs a lot more context and a lot more structure context and that feedback loop. The other big important piece of this is as we were describing earlier, that reallocation, seeing what happens with that reallocation and then feeding it back to the LLM as well as actual human feedback. We're actually telling the model what went well, what didn't go well, right? With that reallocation. So building that system and feeding it back into the model and a couple of other ways. Seeing an improvement where you went from something that just wasn't working to something that's actually producing usable results that have real impact on ⁓ marketing spend and effectiveness.
speaker-1: Love the examples and all the way from ontology to how do we go faster with multiple LLMs, which we can dig into on a future episode excited about the progress the team's made. But I wanted to ask you as we close is AI in this current era of marketing, it's all the buzz, obviously. What do you, what excites you and what are you seeing as you look over the hedge at what's, what's ahead of us?
speaker-0: I feel like agentic and agents, these agentic loops that were going in that direction, but they weren't really fully there. Now, I almost think of the agentic workflows and these capabilities as sort of like a new modality of machine learning. So as we were pre-training and then post-training and these big labs that have been doing that and continue to do that and continue to see improvements, we've moved... as well into this new sort of realm of machine learning where the agent loop becomes a new machine learning modality where that continuous feedback. And then this is, think where we've zeroed in at SlingWave is this continuous feedback from these models that feeds back into the model on real world data, right? Like on real world scenarios. That becomes its own essentially machine learning technique. This is something that I'm actually excited about. And I think the next six to 12 months are going to see step change improvement.
speaker-1: Pio, this has been great. Thank you. And for all our listeners out there, thank you for joining us on today's episode of the incremental marketer. Stay tuned for more voices, more lessons, and more proof that incremental moves can drive big growth. If you enjoyed this conversation, don't forget to subscribe, like, and share, and join us for more episodes with the leaders shaping the future of marketing.
