00:00:06 Rob: Hey everyone, I'm Rob May. I'm the CEO of Neurometric.
00:00:10 Cooper: And I'm Cooper, COO here at Neurometric. And this is Inference Time Tactics, a podcast for researchers, builders, and engineers who want to know what's happening in this next era of AI.
00:00:22 Rob: So today we are all in Santa Clara for the AI infrastructure conference. And we're recording this as part of this conference. And so we are joined today, Cooper and I are by our other two co-founders, Dave Rauchwerk, who leads product and Byron Galbraith, who's our CTO. And we're going to talk about some product updates that we have coming out on the inference side for our inference time compute studio. They can allow you to drag and drop and test a lot of different model and inference time algorithm configurations.
00:00:52 Cooper: Yeah. And so before we begin, maybe we should start with, Dave, you introducing yourself. It's your first time on the podcast. Got a really interesting background.
00:01:01 Dave: Yeah, absolutely. So I have been working with computers and systems for my entire adult life. I started with Linux kernel 2.x in the 90s and over the years I've worked across a number of different domains. Started a number of companies. Most recently, I started a company called the Next Thing Co. that made the world's first $9 computer, something called Pocket CHIP that was also a really interesting open source hardware and software project. And we contributed a lot of things to the Linux kernel. And was actually a really early exposure to some audio based artificial intelligence projects, specifically around sound source localization and wakeboard detection. And it's been really interesting to see this whole world emerge where those of us that have been working across the entire stack are now finally getting the chance to sort of flex our muscles.
00:02:00 Cooper: Yeah, and it's a pretty interesting team composition. I know, Rob, you were really intentional when bringing us together. Do you want to talk about why this blend of experts and how we have a unique advantage in this space?
00:02:12 Rob: Yeah, I mean, two things. Number one, just selfishly being a founder and being a CEO and working with other people that have started and run companies, it just makes my life a lot easier. Because you guys can do a lot of stuff, very self-directed and all that. I think that'll actually scale really well as the company grows. But then also in terms of where AI is going, I think everybody's been very focused on models as the thing. And I think we share a belief that intelligence is moving up to the systems layer. People are going to design AI systems and a very systems focused team in terms of Byron and Dave being the two more technical people have worked at different layers of the stack.
00:02:52 Rob: I, despite running software companies, was originally trained as a hardware engineer. So that's kind of like, actually my point of view on the world is a little more like systems oriented because… doing hardware is very different than doing software because you think about everything happening at the same time and you think about event driven things rather than sequential recipes that you do when you think sometimes when you write code. And then adding you to the team, like for a business person, you're actually very technical, even though you're the least technical person here. And so that… you get these kinds of business models and things that we're doing. Yeah. I think we got a really cool team for where the AI world's going.
00:03;29 Cooper: Yeah, it's fun. And a lot of our early conversations months ago, we're seeing the space catch up. We talked a few episodes ago about GPT-5 and its implementation of ITC strategies and what that signals for the market. I know NVIDIA last week just blogged about a heterogeneous world and the world that SLMs are going to play. And so how do we think about that and positioning the company and the problems we're solving?
00:03:55 Rob: Yeah. I mean, I think we're trying to make it easy for people to understand and evaluate their architectures in that world. So I think this idea that Dave's come up with for the drag and drop interface – I mean, why don't you talk a little bit about [inaudible], Dave? I think it's cool.
00:04:16 Dave: It's really interesting. I mean, this thread of moving from a single model to systems is, I think, something that's becoming really, really important. Anybody who's building applications is starting to realize that a single model is not enough to solve the problem, both in terms of performance, in terms of cost, or even in terms of accuracy and some of the stuff that we've seen in the work that we've been doing. And the challenge is trying to figure out which models to use when and where.
00:04:45 Dave: And I think one of the things that we're really focused on is trying to figure out is there some way that we can help people answer these questions? And is there some way that we can get the community involved in mixing and matching all these different components? There's this incredible explosion of models that have become available through Hugging Face and research and all the different labs working on these different things. But having to actually wire them all up individually to each other and compose them into something that you can actually create a qualitative and ultimately quantitative assessment of the performance for a given domain is like a lot of work.
00:05:21 Dave: And I think in my own work playing with these different models and building applications, I just reach for the highest performance model I can get my hands on. And I think what we're seeing when we talk to customers is that, yeah, that works up to a point. But as soon as you start to scale, it becomes prohibitively expensive. And sometimes there are limits on timeouts and other things that Byron and I have talked about that prevent you from just going with that naive, most powerful model all the time strategy.
00:05:47 Dave: And so I think for us, like the idea that one of the first things we're excited to share is the ability to start with a collection of different templates of AI systems that can be composed of different types of models, large language models, open source models, small language models, reasoning models, all these different things and glue them together in a way where without having to write any code, you can actually see which of these configurations and which frameworks are actually going to be the most useful for your particular application.
00:06:16 Cooper: Absolutely. And to start with this release, what kind of options are people going to have? What can they expect?
00:06:23 Dave: So we're trying to get something out as quickly as possible, naturally. And the first thing is very much read-only, right? Our goal is to, I think, educate people and get them from zero to one on, well, what does an inference time compute strategy look like? And how do you use inference time compute to improve the performance of a given set of open source models? So the thing we're looking to launch really, really soon is the ability to take a pre-cooked set of different AI systems that we're going to give to people and have a limited number of options that you can play with and experiment with and ultimately perform qualitative and quantitative evaluations on these different strategies that we create for folks.
00:07:01 Dave: I think in the future, as we've talked about, if people find this useful, it'd be really, really great to create a way where people could build their own strategies and actually drag and drop and play with things and get to a place where there's a truly composable environment for inference time compute. But I think right now in my own experience with it is you kind of have to get to the place where you fully understand and appreciate, yeah, if I use more compute, I can get better results. Or if I change from large language models to small language models in certain contexts, I can actually get a more performant result for less money. And I think that's… when you hear that the first time, it's almost too good to be true. It seems impossible. And I think creating a place where people can see some of these demonstrations without having to write any code or deploy any servers or rent any GPUs, I think will be really exciting.
00:07:48 Cooper: Absolutely. And bringing that to life, there are a lot of engineering challenges as we look at ITC engine and then this GUI front end. Do you want to talk about some of that?
00:08:01 Byron: Yeah, so again, we have multiple components here. Not only do you have the front end, which has to give you the view of all of the different components that Dave was talking about, but also we have to then instrument that on the back. And part of what we've been doing in the past, and we've talked about this, is around the challenge of really running some of these evaluations at any kind of scale to be able to do these sorts of experiments. And so we're leveraging that initial work to then make it so that you can then run these sorts of things on demand in a way that is going to be more effective and productive for people.
00:08:35 Byron: Because again, the goal here is to be able to see these things. As Dave mentioned, it's one thing to go and look at and say, I want to use a Hugging Face model or look at an API recipe and you can wire that up and be cool. Okay, I got a little demo working. It's entirely different to be able to run this in a production environment at any kind of scale or for any kind of load. I mean, just even initial experiments we've seen, you can very quickly run into issues with third party providers with their own rate limiting. And so maybe something that works well for you in a small proof of concept or demo scale type of situation will very, very quickly fall over if you try to turn it on for any sort of volume. And so part of what we're doing here is also figuring out how to make that smooth.
00:09:22 Rob: Yeah. I think what's interesting to me about it is that we're still in this part of the market where the vast majority of people are just trying the big models and they're just trying to see what the capabilities are and how they build those into whatever systems they're building or whatever. But the more advanced companies we've seen, the ones that have been on AI models for a couple of years are all following the same pattern, which is they start with one giant model, something that's typically state of the art. And then they start to add other models for a couple of different reasons. Sometimes because, to your point on the engineering challenges of the models, they can't get enough access, they have API issues or they can't get the token throughput they want and they just need some other options.
00:10:00 Rob: Sometimes because they realize that like, well, they actually don't need that capability of model for everything they're doing. And so they start to break off in pieces, small language models and other things like that. And so I just… as a long time technology entrepreneur, I just see that as a trend that probably everybody's going to go through over the next decade. And so I think it's something good to lean into, which is part of why I'm excited that we're building this.
00:10:21 Dave: It definitely seems like there is an explosion in complexity where there's just so many choices, so many different frameworks, so many different models. It continues to grow every single day. And the thing I keep reaching for is, okay, like it's Monday or it's Tuesday. What's come out, right? And how do I know if the thing that I'm using is actually any good? For me, it's really fascinating because… like we talked about GPT-5 earlier, right? And this idea that GPT-5 is not just a single model, right? As we understand it, and what we can see of it, it's got a router, it's got multiple different models, it's really a system, right?
00:11:02 Dave: And I think, as we talked earlier today, it was really very interesting to think about what happens when the ability to compose these different models and have the right amount of compute for the right task at the right time in a way that people can use out of the box and incorporate open models into that in a configurable way – that to me as a developer, as a user is actually really interesting and very exciting because I feel as powerful and capable as GPT-5 is, I don't really have any handles that I can change. I want to change the way that it works. I'm going to customize it.
00:11:34 Dave: So it's really, really interesting to see even the shift in the benchmarking where I think thinking about the move from a single model to, okay, well, what is actually the performance of the whole system and where can you optimize the different things that I think is… is going to – today, It feels really novel. I think as you're talking about over the next couple of years, this is how everyone's going to be doing it.
00:11:55 Rob: Do you think it's weird? I mean, you mentioned a couple of minutes ago that there's this explosion in tools. Don't you think it's a little bit weird compared to previous technology ways that we haven't started to standardize? There is no standard AI stack. There's not even anything close from what I can tell. And it's like… going back to when I got into tech, like the LAMP stack was kind of the thing and while there's always this explosion in the beginning with no SQL databases and with web frameworks and stuff like that, they kind of coalesce around a couple of different use cases. You got like Mongo and React and Cassandra and things like that. And we just haven't seen that yet.
00:12:32 Dave: I agree completely. It's really fascinating as somebody who's been involved in a lot of open source communities to see how it… fractals of innovation as they happen. I will point out that I think we're having a moment right now for the terminal user interface that has, like, I would have never thought was going to happen. I'm a huge fan of that as a Linux person forever. Like the TUI is back and it's the way you do things. It makes me really happy. It's awesome to see. And I feel like that… if there is a common thread, it is… the most traction I've seen, the most success, the most standardization I've seen has been at the protocol layer. I think the MCP stuff was really, really… it's been really, really interesting. And we were talking about that earlier where no matter how you feel about its implementation, it's evolving so fast and that everybody seems to be agreeing and coalescing around it.
00:13:16 Dave: Like I feel like the frameworks, everyone's just sort of reaching for pretty much whatever they can get to work. And when you mix that in with the emergence of Claude Code or Codex, you also see this thing happen where somebody builds something in Python and then someone transpiles it using generative AI into something in JavaScript. And so it's almost like, do you even care what framework you're using? It's more like, does it work at all? Right? If you could build something that works, it's such a fascinating moment where we've reached this era where the actual cost of building the software itself and the frameworks, it's fallen to effectively zero.
00:13:54 Dave: And I know we've talked a lot about this and I actually enjoyed your post about the LAMP stack. Are we going to be quickly entering into a place where you know what? There is no framework. You just generate it for whatever project you're doing. I don't know if we're there yet, but the idea where the standard is that… we're all basically making a lot more software than we were before.
00:14:17 Byron: Well, actually, to take it in a slightly different direction, I think that one of the things we're seeing, had a colleague who had mentioned this, is that we are in that sort of early Web 1.0 era, right? Where it's like, how do I build a website? What is the right way to do it? How do I build a database driven application on the web? What does it even mean? Right. And so LAMP was a way that really came out, was a popular solution for people who couldn't pay for like the really expensive.
00:14:45 Rob: We're in the cold fusion phase. [crosstalk]
00:14:48 Byron: Exactly. It's like we've got… there was a time right when the things were like, okay, it's IIS and ASP or it's these really heavyweight Java frameworks or it's a cold fusion is sort of the slightly more business friendly sort of weird package thing. So lots of different things are out there because no one really knew how to do this. They're just grabbing for anything because “got to have a website,” right? So I think, it's like that same idea that “got to have AI. I don't know what works or why or how.” And that's why I think we're seeing this sort of just nobody really has standardized yet.
00:15:22 Rob: Well, and what will happen as all this stuff gets out there is… part of the problem is AI has so many use cases, right? When you're talking about replacing intelligence and actual work, it's so different than the kinds of technology that came before it. And so that variability I think is part of the reason you're seeing the fragmentation. And then what'll happen is as more and more use cases emerge and people start to see patterns – because you still… I mean, you could kind of say RAG is a pattern, but the truth is so many people that you talk to have all these different nuances in their system because work is so different in so many places.
00:15:57 Rob: But as we start to see buckets of things that consolidate the frameworks and the stacks and everything will sort of consolidate economically around those, right? What's the most economically viable way to do some of this stuff? And that's kind of how I expect it to play out. And it's part of why I like this stuff that we're building, which allows you to experiment with what works best for you. And then hopefully we can translate that further down the workflow path for you at some point.
00:16:26 Dave: It definitely feels like we're… in the conversation we have about the architecture of the things that we're building and how we're trying to put things together. We are trying to create sandboxes that allow you to encourage more sharing and more exchanging and to create some standardization. Right. I mean, because ultimately we want to compare apples to apples. And at the end of the day, it's really cool to see so many different benchmarks, so many different kinds of models but it seems that – especially when you get into the realm of systems, it's really, really hard to say, I have AI system A and AI system B and which one is better and for what and how, right?
00:17:00 Dave: It's your point about where we are in the standardization with respect to the technology history… it's interesting to see. I don't know that it's going to be winner take all with all the stacks. Right. And it's good. Fascinating to see how it emerges. It's also really fascinating to hear… just how much more competitive the open source models have become versus the commercial ones. And if you had asked me, we talked about this like, I don't know, six months ago or a year ago for sure, I would not have had the same amount of confidence that I have in the open source stack. And I don't think I would have been as excited about the idea of, okay, we could probably glue together a number of different open source models and technologies to get something that approaches the performance and competitiveness of a commercial solution. It's a very exciting time.
00:17:52 Rob: Well, some of that's because, pre-training is slowed down. And so that's part of why that's happened. But yeah, it's a good point.
00:18:03 Cooper: And open source is a big part of our strategy. So we're shipping this read-only today, but how do we want people to engage with this studio? What kind of feedback do we want and what can we expect about community engagement in the future?
00:18:17 Dave: I think the most interesting questions are what kind of benchmarks do people want to perform? And I think we've seen a lot of that already in the open ecosystem, a lot of innovation, everything from the Terminal-Bench to CRMArena, which we've been working on and a lot of really interesting domain-specific benchmarks, those like in accounting and running small companies and medical QA and all these sorts of things. But I think that… what are the gaps? What are the missing benchmarks? What do people want to see?
00:18:45 Dave: And then on the AI system side, like what are beyond the models? What are the other primitives, the other components that people want to incorporate into their production AI systems? We talked about RAG patterns. We've talked about graph databases. We've talked about key value stores. We talked about the ability to have some kind of shared memory, context across sessions. We have a lot of ideas about what these things are, but given that we're still a young company, we can't do them all at once. And so setting the priorities, I think, is something that we really need feedback to determine. What should we build first?
00:19:18 Dave: And I think in the first cut, it's very much about, I think, distilling what we see in the market into a tool that people can, I think, appreciate the way that we understand how these things work. And I think we can get… approach a level of clarity where we can all, I think, have a conversation with shared vocabulary. I think that's the ultimate goal.
00:19:39 Cooper: Absolutely. So we had a lot of exciting conversation today. Not sure how much we can share on the podcast, but anything else we want to touch on before we wrap.
00:19:49 Rob: No, just I think by the time people see this, they should go to neurometric.ai and check stuff out, send us some feedback. And really, if you're working on a really interesting inference problem, or you're doing a lot of inference at scale that you need shorter latency or lower cost or your inference needs vary pretty widely… I mean, I think we'd love to talk to you. That'd be pretty interesting to work with.
00:20:17 Cooper: Absolutely. Well, that's it for today. Thanks for listening to Inference Time Tactics. Like subscribe, share, and looking forward to talking to you again. Thanks.
00:00:10 Cooper: And I'm Cooper, COO here at Neurometric. And this is Inference Time Tactics, a podcast for researchers, builders, and engineers who want to know what's happening in this next era of AI.
00:00:22 Rob: So today we are all in Santa Clara for the AI infrastructure conference. And we're recording this as part of this conference. And so we are joined today, Cooper and I are by our other two co-founders, Dave Rauchwerk, who leads product and Byron Galbraith, who's our CTO. And we're going to talk about some product updates that we have coming out on the inference side for our inference time compute studio. They can allow you to drag and drop and test a lot of different model and inference time algorithm configurations.
00:00:52 Cooper: Yeah. And so before we begin, maybe we should start with, Dave, you introducing yourself. It's your first time on the podcast. Got a really interesting background.
00:01:01 Dave: Yeah, absolutely. So I have been working with computers and systems for my entire adult life. I started with Linux kernel 2.x in the 90s and over the years I've worked across a number of different domains. Started a number of companies. Most recently, I started a company called the Next Thing Co. that made the world's first $9 computer, something called Pocket CHIP that was also a really interesting open source hardware and software project. And we contributed a lot of things to the Linux kernel. And was actually a really early exposure to some audio based artificial intelligence projects, specifically around sound source localization and wakeboard detection. And it's been really interesting to see this whole world emerge where those of us that have been working across the entire stack are now finally getting the chance to sort of flex our muscles.
00:02:00 Cooper: Yeah, and it's a pretty interesting team composition. I know, Rob, you were really intentional when bringing us together. Do you want to talk about why this blend of experts and how we have a unique advantage in this space?
00:02:12 Rob: Yeah, I mean, two things. Number one, just selfishly being a founder and being a CEO and working with other people that have started and run companies, it just makes my life a lot easier. Because you guys can do a lot of stuff, very self-directed and all that. I think that'll actually scale really well as the company grows. But then also in terms of where AI is going, I think everybody's been very focused on models as the thing. And I think we share a belief that intelligence is moving up to the systems layer. People are going to design AI systems and a very systems focused team in terms of Byron and Dave being the two more technical people have worked at different layers of the stack.
00:02:52 Rob: I, despite running software companies, was originally trained as a hardware engineer. So that's kind of like, actually my point of view on the world is a little more like systems oriented because… doing hardware is very different than doing software because you think about everything happening at the same time and you think about event driven things rather than sequential recipes that you do when you think sometimes when you write code. And then adding you to the team, like for a business person, you're actually very technical, even though you're the least technical person here. And so that… you get these kinds of business models and things that we're doing. Yeah. I think we got a really cool team for where the AI world's going.
00:03;29 Cooper: Yeah, it's fun. And a lot of our early conversations months ago, we're seeing the space catch up. We talked a few episodes ago about GPT-5 and its implementation of ITC strategies and what that signals for the market. I know NVIDIA last week just blogged about a heterogeneous world and the world that SLMs are going to play. And so how do we think about that and positioning the company and the problems we're solving?
00:03:55 Rob: Yeah. I mean, I think we're trying to make it easy for people to understand and evaluate their architectures in that world. So I think this idea that Dave's come up with for the drag and drop interface – I mean, why don't you talk a little bit about [inaudible], Dave? I think it's cool.
00:04:16 Dave: It's really interesting. I mean, this thread of moving from a single model to systems is, I think, something that's becoming really, really important. Anybody who's building applications is starting to realize that a single model is not enough to solve the problem, both in terms of performance, in terms of cost, or even in terms of accuracy and some of the stuff that we've seen in the work that we've been doing. And the challenge is trying to figure out which models to use when and where.
00:04:45 Dave: And I think one of the things that we're really focused on is trying to figure out is there some way that we can help people answer these questions? And is there some way that we can get the community involved in mixing and matching all these different components? There's this incredible explosion of models that have become available through Hugging Face and research and all the different labs working on these different things. But having to actually wire them all up individually to each other and compose them into something that you can actually create a qualitative and ultimately quantitative assessment of the performance for a given domain is like a lot of work.
00:05:21 Dave: And I think in my own work playing with these different models and building applications, I just reach for the highest performance model I can get my hands on. And I think what we're seeing when we talk to customers is that, yeah, that works up to a point. But as soon as you start to scale, it becomes prohibitively expensive. And sometimes there are limits on timeouts and other things that Byron and I have talked about that prevent you from just going with that naive, most powerful model all the time strategy.
00:05:47 Dave: And so I think for us, like the idea that one of the first things we're excited to share is the ability to start with a collection of different templates of AI systems that can be composed of different types of models, large language models, open source models, small language models, reasoning models, all these different things and glue them together in a way where without having to write any code, you can actually see which of these configurations and which frameworks are actually going to be the most useful for your particular application.
00:06:16 Cooper: Absolutely. And to start with this release, what kind of options are people going to have? What can they expect?
00:06:23 Dave: So we're trying to get something out as quickly as possible, naturally. And the first thing is very much read-only, right? Our goal is to, I think, educate people and get them from zero to one on, well, what does an inference time compute strategy look like? And how do you use inference time compute to improve the performance of a given set of open source models? So the thing we're looking to launch really, really soon is the ability to take a pre-cooked set of different AI systems that we're going to give to people and have a limited number of options that you can play with and experiment with and ultimately perform qualitative and quantitative evaluations on these different strategies that we create for folks.
00:07:01 Dave: I think in the future, as we've talked about, if people find this useful, it'd be really, really great to create a way where people could build their own strategies and actually drag and drop and play with things and get to a place where there's a truly composable environment for inference time compute. But I think right now in my own experience with it is you kind of have to get to the place where you fully understand and appreciate, yeah, if I use more compute, I can get better results. Or if I change from large language models to small language models in certain contexts, I can actually get a more performant result for less money. And I think that's… when you hear that the first time, it's almost too good to be true. It seems impossible. And I think creating a place where people can see some of these demonstrations without having to write any code or deploy any servers or rent any GPUs, I think will be really exciting.
00:07:48 Cooper: Absolutely. And bringing that to life, there are a lot of engineering challenges as we look at ITC engine and then this GUI front end. Do you want to talk about some of that?
00:08:01 Byron: Yeah, so again, we have multiple components here. Not only do you have the front end, which has to give you the view of all of the different components that Dave was talking about, but also we have to then instrument that on the back. And part of what we've been doing in the past, and we've talked about this, is around the challenge of really running some of these evaluations at any kind of scale to be able to do these sorts of experiments. And so we're leveraging that initial work to then make it so that you can then run these sorts of things on demand in a way that is going to be more effective and productive for people.
00:08:35 Byron: Because again, the goal here is to be able to see these things. As Dave mentioned, it's one thing to go and look at and say, I want to use a Hugging Face model or look at an API recipe and you can wire that up and be cool. Okay, I got a little demo working. It's entirely different to be able to run this in a production environment at any kind of scale or for any kind of load. I mean, just even initial experiments we've seen, you can very quickly run into issues with third party providers with their own rate limiting. And so maybe something that works well for you in a small proof of concept or demo scale type of situation will very, very quickly fall over if you try to turn it on for any sort of volume. And so part of what we're doing here is also figuring out how to make that smooth.
00:09:22 Rob: Yeah. I think what's interesting to me about it is that we're still in this part of the market where the vast majority of people are just trying the big models and they're just trying to see what the capabilities are and how they build those into whatever systems they're building or whatever. But the more advanced companies we've seen, the ones that have been on AI models for a couple of years are all following the same pattern, which is they start with one giant model, something that's typically state of the art. And then they start to add other models for a couple of different reasons. Sometimes because, to your point on the engineering challenges of the models, they can't get enough access, they have API issues or they can't get the token throughput they want and they just need some other options.
00:10:00 Rob: Sometimes because they realize that like, well, they actually don't need that capability of model for everything they're doing. And so they start to break off in pieces, small language models and other things like that. And so I just… as a long time technology entrepreneur, I just see that as a trend that probably everybody's going to go through over the next decade. And so I think it's something good to lean into, which is part of why I'm excited that we're building this.
00:10:21 Dave: It definitely seems like there is an explosion in complexity where there's just so many choices, so many different frameworks, so many different models. It continues to grow every single day. And the thing I keep reaching for is, okay, like it's Monday or it's Tuesday. What's come out, right? And how do I know if the thing that I'm using is actually any good? For me, it's really fascinating because… like we talked about GPT-5 earlier, right? And this idea that GPT-5 is not just a single model, right? As we understand it, and what we can see of it, it's got a router, it's got multiple different models, it's really a system, right?
00:11:02 Dave: And I think, as we talked earlier today, it was really very interesting to think about what happens when the ability to compose these different models and have the right amount of compute for the right task at the right time in a way that people can use out of the box and incorporate open models into that in a configurable way – that to me as a developer, as a user is actually really interesting and very exciting because I feel as powerful and capable as GPT-5 is, I don't really have any handles that I can change. I want to change the way that it works. I'm going to customize it.
00:11:34 Dave: So it's really, really interesting to see even the shift in the benchmarking where I think thinking about the move from a single model to, okay, well, what is actually the performance of the whole system and where can you optimize the different things that I think is… is going to – today, It feels really novel. I think as you're talking about over the next couple of years, this is how everyone's going to be doing it.
00:11:55 Rob: Do you think it's weird? I mean, you mentioned a couple of minutes ago that there's this explosion in tools. Don't you think it's a little bit weird compared to previous technology ways that we haven't started to standardize? There is no standard AI stack. There's not even anything close from what I can tell. And it's like… going back to when I got into tech, like the LAMP stack was kind of the thing and while there's always this explosion in the beginning with no SQL databases and with web frameworks and stuff like that, they kind of coalesce around a couple of different use cases. You got like Mongo and React and Cassandra and things like that. And we just haven't seen that yet.
00:12:32 Dave: I agree completely. It's really fascinating as somebody who's been involved in a lot of open source communities to see how it… fractals of innovation as they happen. I will point out that I think we're having a moment right now for the terminal user interface that has, like, I would have never thought was going to happen. I'm a huge fan of that as a Linux person forever. Like the TUI is back and it's the way you do things. It makes me really happy. It's awesome to see. And I feel like that… if there is a common thread, it is… the most traction I've seen, the most success, the most standardization I've seen has been at the protocol layer. I think the MCP stuff was really, really… it's been really, really interesting. And we were talking about that earlier where no matter how you feel about its implementation, it's evolving so fast and that everybody seems to be agreeing and coalescing around it.
00:13:16 Dave: Like I feel like the frameworks, everyone's just sort of reaching for pretty much whatever they can get to work. And when you mix that in with the emergence of Claude Code or Codex, you also see this thing happen where somebody builds something in Python and then someone transpiles it using generative AI into something in JavaScript. And so it's almost like, do you even care what framework you're using? It's more like, does it work at all? Right? If you could build something that works, it's such a fascinating moment where we've reached this era where the actual cost of building the software itself and the frameworks, it's fallen to effectively zero.
00:13:54 Dave: And I know we've talked a lot about this and I actually enjoyed your post about the LAMP stack. Are we going to be quickly entering into a place where you know what? There is no framework. You just generate it for whatever project you're doing. I don't know if we're there yet, but the idea where the standard is that… we're all basically making a lot more software than we were before.
00:14:17 Byron: Well, actually, to take it in a slightly different direction, I think that one of the things we're seeing, had a colleague who had mentioned this, is that we are in that sort of early Web 1.0 era, right? Where it's like, how do I build a website? What is the right way to do it? How do I build a database driven application on the web? What does it even mean? Right. And so LAMP was a way that really came out, was a popular solution for people who couldn't pay for like the really expensive.
00:14:45 Rob: We're in the cold fusion phase. [crosstalk]
00:14:48 Byron: Exactly. It's like we've got… there was a time right when the things were like, okay, it's IIS and ASP or it's these really heavyweight Java frameworks or it's a cold fusion is sort of the slightly more business friendly sort of weird package thing. So lots of different things are out there because no one really knew how to do this. They're just grabbing for anything because “got to have a website,” right? So I think, it's like that same idea that “got to have AI. I don't know what works or why or how.” And that's why I think we're seeing this sort of just nobody really has standardized yet.
00:15:22 Rob: Well, and what will happen as all this stuff gets out there is… part of the problem is AI has so many use cases, right? When you're talking about replacing intelligence and actual work, it's so different than the kinds of technology that came before it. And so that variability I think is part of the reason you're seeing the fragmentation. And then what'll happen is as more and more use cases emerge and people start to see patterns – because you still… I mean, you could kind of say RAG is a pattern, but the truth is so many people that you talk to have all these different nuances in their system because work is so different in so many places.
00:15:57 Rob: But as we start to see buckets of things that consolidate the frameworks and the stacks and everything will sort of consolidate economically around those, right? What's the most economically viable way to do some of this stuff? And that's kind of how I expect it to play out. And it's part of why I like this stuff that we're building, which allows you to experiment with what works best for you. And then hopefully we can translate that further down the workflow path for you at some point.
00:16:26 Dave: It definitely feels like we're… in the conversation we have about the architecture of the things that we're building and how we're trying to put things together. We are trying to create sandboxes that allow you to encourage more sharing and more exchanging and to create some standardization. Right. I mean, because ultimately we want to compare apples to apples. And at the end of the day, it's really cool to see so many different benchmarks, so many different kinds of models but it seems that – especially when you get into the realm of systems, it's really, really hard to say, I have AI system A and AI system B and which one is better and for what and how, right?
00:17:00 Dave: It's your point about where we are in the standardization with respect to the technology history… it's interesting to see. I don't know that it's going to be winner take all with all the stacks. Right. And it's good. Fascinating to see how it emerges. It's also really fascinating to hear… just how much more competitive the open source models have become versus the commercial ones. And if you had asked me, we talked about this like, I don't know, six months ago or a year ago for sure, I would not have had the same amount of confidence that I have in the open source stack. And I don't think I would have been as excited about the idea of, okay, we could probably glue together a number of different open source models and technologies to get something that approaches the performance and competitiveness of a commercial solution. It's a very exciting time.
00:17:52 Rob: Well, some of that's because, pre-training is slowed down. And so that's part of why that's happened. But yeah, it's a good point.
00:18:03 Cooper: And open source is a big part of our strategy. So we're shipping this read-only today, but how do we want people to engage with this studio? What kind of feedback do we want and what can we expect about community engagement in the future?
00:18:17 Dave: I think the most interesting questions are what kind of benchmarks do people want to perform? And I think we've seen a lot of that already in the open ecosystem, a lot of innovation, everything from the Terminal-Bench to CRMArena, which we've been working on and a lot of really interesting domain-specific benchmarks, those like in accounting and running small companies and medical QA and all these sorts of things. But I think that… what are the gaps? What are the missing benchmarks? What do people want to see?
00:18:45 Dave: And then on the AI system side, like what are beyond the models? What are the other primitives, the other components that people want to incorporate into their production AI systems? We talked about RAG patterns. We've talked about graph databases. We've talked about key value stores. We talked about the ability to have some kind of shared memory, context across sessions. We have a lot of ideas about what these things are, but given that we're still a young company, we can't do them all at once. And so setting the priorities, I think, is something that we really need feedback to determine. What should we build first?
00:19:18 Dave: And I think in the first cut, it's very much about, I think, distilling what we see in the market into a tool that people can, I think, appreciate the way that we understand how these things work. And I think we can get… approach a level of clarity where we can all, I think, have a conversation with shared vocabulary. I think that's the ultimate goal.
00:19:39 Cooper: Absolutely. So we had a lot of exciting conversation today. Not sure how much we can share on the podcast, but anything else we want to touch on before we wrap.
00:19:49 Rob: No, just I think by the time people see this, they should go to neurometric.ai and check stuff out, send us some feedback. And really, if you're working on a really interesting inference problem, or you're doing a lot of inference at scale that you need shorter latency or lower cost or your inference needs vary pretty widely… I mean, I think we'd love to talk to you. That'd be pretty interesting to work with.
00:20:17 Cooper: Absolutely. Well, that's it for today. Thanks for listening to Inference Time Tactics. Like subscribe, share, and looking forward to talking to you again. Thanks.
