UTSKRIFT 🔗
WEBVTT
00:00:00.239 --> 00:00:13.119
The most common agent like multi-agent setup that I see today is I'll have one agent that I control doing the implementation of a particular feature or fixing a bug or something like that.
00:00:13.279 --> 00:00:17.039
And then another agent that my team controls does the code review on that.
00:00:17.280 --> 00:00:21.839
And like, you know, my team controls the prompts for the code review, I control the prompt that drives the implementation.
00:00:22.320 --> 00:00:29.519
Welcome to First Commit, a podcast from Notable Capital where we talk to the builders solving hard problems before the industry catches up.
00:00:31.280 --> 00:00:41.280
So today we have Robert Brennan, who is the CEO and co-founder of Open Hands, which is an open source research project that became one of the most starred AI coding agent repos on GitHub.
00:00:41.439 --> 00:00:42.159
Welcome, Robert.
00:00:42.479 --> 00:00:42.719
Thanks.
00:00:42.880 --> 00:00:43.520
Thanks for having me.
00:00:43.840 --> 00:00:44.159
Awesome.
00:00:44.320 --> 00:00:45.439
Excited to hop in.
00:00:45.679 --> 00:00:55.840
So most people who have become technical over their life has like a specific moment or something that revealed kind of how they thought about coding or what it meant to write code.
00:00:56.000 --> 00:00:59.200
I'm just curious, what did that feel like or how does that matter to you?
00:00:59.600 --> 00:01:07.599
I would say, you know, when I took my my first programming class in college, I wasn't yet sure if I wanted to be a software engineer, but I wanted to dip my toes in the water.
00:01:07.680 --> 00:01:09.359
I had some friends that were CS majors.
00:01:09.519 --> 00:01:14.879
I was originally a math major and thought that that's what I wanted to do, but did a class in Java.
00:01:15.040 --> 00:01:19.200
And for the like final project, we kind of get to pick whatever, whatever we wanted to do.
00:01:19.280 --> 00:01:21.920
And I built a graphing calculator from scratch in Java.
00:01:22.000 --> 00:01:23.840
And it really felt like I had superpowers.
00:01:23.920 --> 00:01:26.879
Like I could suddenly make this computer do whatever I wanted it to do.
00:01:26.959 --> 00:01:28.719
I could make it display what I wanted it to.
00:01:28.799 --> 00:01:31.680
I like I knew, you know, I was still very much a baby programmer.
00:01:31.760 --> 00:01:33.359
I didn't really know what I was doing.
00:01:33.519 --> 00:01:34.480
My code was a mess.
00:01:34.640 --> 00:01:40.959
But like I really felt this feeling of having like magic powers that like I could I could do whatever I wanted with the computer.
00:01:41.120 --> 00:01:45.519
And that was that was a very big unlock for me, was just that that feeling of power.
00:01:45.680 --> 00:01:54.719
And I would say over the course of you know, the next 15 years of my career, that became much more mundane and it became very rote and it became less fun, less magical.
00:01:54.879 --> 00:01:55.760
It became a job, right?
00:01:56.000 --> 00:02:06.000
But what's really fun is that over the last year or two, as AI coding has emerged again, I feel that that feeling again of having magic powers of being able to just like create things out of the blue.
00:02:06.239 --> 00:02:09.199
And it's it's really like uh reinvigorated my love for programming.
00:02:09.520 --> 00:02:20.000
Do you think you would have had that initial unlock having the tools we had now, like 10 years ago, just given that things are so easy now and sometimes it's less thinking and more doing?
00:02:20.319 --> 00:02:24.159
Yeah, I think there's still a lot of like creativity and problem solving involved.
00:02:24.319 --> 00:02:26.319
It's happening at another level of abstraction.
00:02:26.400 --> 00:02:35.439
But um, you know, while I can, you know, prompt an AI to get me a static website that looks pretty good really quickly, it's still pretty tricky to build a good user experience.
00:02:35.599 --> 00:02:45.280
You know, there's a lot of like turning the loop with the AI and iterating and moving forward to something that, you know, looks nice, works well for the user, that's technically feasible.
00:02:45.520 --> 00:02:56.879
I think even though we've added a lot of you know jet fuel to the thing here, like and we can go a lot faster and do a lot more with fewer people, there's still a lot of creativity and critical thinking involved in creating software.
00:02:57.120 --> 00:02:59.360
I still certainly get that same, that same boost.
00:02:59.439 --> 00:03:05.520
And I imagine folks that are just coming up and just starting to get into software also feel that same creative boost.
00:03:05.840 --> 00:03:06.560
Yeah, for sure.
00:03:06.719 --> 00:03:11.919
And I think you've brought that experience and intuitive UI into OpenDev and Open Hands.
00:03:12.159 --> 00:03:19.120
It would be great to just touch on how you thought about building this initial project, which is now a company, and what that process was like.
00:03:19.439 --> 00:03:22.639
It's been a really tricky problem because it is such a new technology.
00:03:22.800 --> 00:03:27.759
I would say there are we're still experimenting with patterns for human-agent interaction.
00:03:28.000 --> 00:03:34.159
I think one of the biggest deciding points for us as a company was early on as cursors started to blow up.
00:03:34.319 --> 00:03:40.400
Um, we were like, hey, should we actually ditch what we're doing right now and start building an IDE because IDEs are all over the place right now.
00:03:40.639 --> 00:03:45.680
And we made the hard decision that no, we think the IDE is is an old, old form factor.
00:03:45.759 --> 00:03:46.800
We think it's going away.
00:03:46.960 --> 00:03:50.000
We think folks need a new interface for managing agents.
00:03:50.159 --> 00:03:52.479
And I would say ultimately that has borne out, right?
00:03:52.560 --> 00:03:53.919
Folks have moved away from IDEs.
00:03:54.000 --> 00:03:55.919
They're
Load full transcript
now, I would say, working with CLIs.
00:03:56.159 --> 00:04:04.240
Generally speaking, though, we've seen developers follow an adoption curve where they're they're making incremental steps from how they used to work to the new world, right?
00:04:04.639 --> 00:04:13.199
You know, they started with autocomplete because that was, you know, they've been using autocomplete for decades, just got a million times better, but they already had that muscle memory of like hit the tab key to autocomplete.
00:04:13.280 --> 00:04:16.720
And then they took, you know, one more step down the road to like AI-powered IDEs.
00:04:16.800 --> 00:04:20.399
They were used to using IDEs, but now there was an A chat bot there with them.
00:04:20.560 --> 00:04:26.160
But now they've they've taken one step further and they're using like local CLIs to manage to manage agents.
00:04:26.240 --> 00:04:29.360
And, you know, they're used to used to working on the command line, things like that.
00:04:29.519 --> 00:04:50.480
But what we've found with open hands is is you know, we're pushing kind of brand new user interfaces for managing multiple agents, for running agents that run on a schedule or in response to events coming out of the cloud, which are it's a much more foreign way for developers to work, but it's much more powerful once you do get into a groove with it, once you do start uh getting used to that new form factor.
00:04:50.720 --> 00:04:52.800
And we think that's that's really where the future sits.
00:04:52.879 --> 00:05:04.800
And the early adopters that we work with and the folks that we see who are at the forefront of AI-driven development have moved beyond IDEs, they've moved beyond CLIs, and they're really working with brand new user interfaces for managing agents at scale.
00:05:05.120 --> 00:05:09.759
And when you say it's more powerful and these developers are at the forefront, what does that look like in practice?
00:05:09.839 --> 00:05:12.480
What types of agents are they working or are they running?
00:05:12.720 --> 00:05:15.120
There's kind of a maturity curve.
00:05:15.439 --> 00:05:22.879
Most developers they'll they'll start by like, you know, typing prompts into a CLI or a chat window to get an agent to do one thing at a time.
00:05:22.959 --> 00:05:31.360
And they might start managing two or three agents at a time, you know, kind of, but still manually prompting, often like, you know, hitting the Y key to give them permission to do things.
00:05:31.519 --> 00:05:39.040
Then I would say the next step along the curve, you know, to going from crawl to walk, is you start to notice, hey, I'm I'm typing the same prompt over and over again.
00:05:39.120 --> 00:05:42.959
Every time I want to do a code review, I'm telling the agent, you know, the same five things to look for.
00:05:43.279 --> 00:05:58.560
So you start building skills and you start building plugins in order to take those parts that you're repeating over and over again and abstract them out into some some bit of context that the agent can dynamically pull in every time it does a code review or every time it does a bug fix, it like it brings in this context.
00:05:58.800 --> 00:06:18.800
But the the really like mature folks, the ones that really are the early adopters here who are getting the most out of AI, they then make the step from walk to run where instead of you know continuing to manually prompt do a code review, fix a bug, they set up an automation where these agents are running either on a schedule or in response to like events coming out of the cloud, right?
00:06:18.879 --> 00:06:33.600
So they might uh set up an agent that every time there's an error level log in Datadog, the agent immediately jumps in, goes and investigates the error, and then submits a change to the code base to either fix the error or downgrade it to a warning log or improve the error handling.
00:06:33.839 --> 00:06:37.279
But the the idea is the engineer no longer needs to like get interrupted.
00:06:37.439 --> 00:06:40.000
The engineer no longer needs to like context switch over to this task.
00:06:40.079 --> 00:06:45.759
The agent's just taking care of it end-to-end, opens up a pull request, and the developer just needs to rubber stamp it and it gets into the code base.
00:06:46.079 --> 00:06:56.399
So you get much a much bigger lift in terms of your capacity to take on a lot of work because these agents are running automatically rather than the developer having to like manually pilot them.
00:06:56.720 --> 00:06:57.120
Got it.
00:06:57.199 --> 00:07:05.839
So it sounds like the goal for OpenEnds is to be more of the go-to agent builder, agent infrastructure platform versus the autocomplete coding co-pilot.
00:07:06.160 --> 00:07:06.480
Exactly.
00:07:06.560 --> 00:07:11.839
Yeah, I would say there's there's really two work streams that we see at companies today.
00:07:12.160 --> 00:07:15.120
One is this very familiar model of like pair programming with AI.
00:07:15.360 --> 00:07:23.600
So that might be a developer using an IDE like cursor or a CLI like Claud Code, but they're working very similarly to how they worked 10 years ago, right?
00:07:23.680 --> 00:07:31.199
They're picking up a tier a ticket out of Jira, they're opening up their IDE or their CLI, and they're squinting at their computer screen and figuring out how am I going to fix this thing.
00:07:31.279 --> 00:07:36.399
And the agent is there to help them go typically about 20 or 30% faster if you look at the most recent studies.
00:07:36.639 --> 00:07:37.920
And that is very valuable.
00:07:38.079 --> 00:07:47.600
It's a very good workflow for specifically like a very ad hoc bespoke work, you know, a very tricky new feature we need to implement or a bug that, you know, nobody's been able to reproduce yet.
00:07:47.920 --> 00:08:01.519
That's the kind of thing where you do want like a quick feedback loop between the human and the agent so that they can kind of iterate their way
towards a solution where the human is bringing in their creativity and critical thinking and the agent's doing the rote work of editing files and hunting around the code base.
00:08:01.680 --> 00:08:05.600
But there's a second work stream where you get a much bigger productivity boost, right?
00:08:05.680 --> 00:08:08.160
You only get 20 to 30% for that pair programming model.
00:08:08.319 --> 00:08:20.480
But when you're able to run agents automatically, uh typically for much more like repeatable, automatable tasks, you can get, you know, literally a 10x productivity boost because you're no longer grabbing an engineer's attention and having them focus on this task.
00:08:20.639 --> 00:08:32.159
So things like managing dependencies, things like cutting release notes, things like doing code reviews, things like you know, responding to errors coming out of Datadog, doing like immediate incident response, you know, ticket triage.
00:08:32.240 --> 00:08:39.759
There are a whole bunch of tasks that are just rote parts of a developer's time that like they can they can very easily offload to agents, right?
00:08:39.840 --> 00:08:46.000
All the all that toil that goes into software engineering, all that like keep the lights on type work that goes into making a code base healthy.
00:08:46.080 --> 00:08:49.519
Um, those are great tasks for agents to take on end-to-end.
00:08:49.679 --> 00:08:53.759
So the developer is freed up to do the more creative, more like critical thinking parts of their job.
00:08:54.000 --> 00:08:54.559
Yeah, for sure.
00:08:54.639 --> 00:09:02.720
And I feel like in that second bucket, you have companies that are focused on specific parts of the development process, whether it's incident management, whether it's PRs, code review, et cetera.
00:09:02.879 --> 00:09:07.360
You have like the no-code agent builders, maybe the more technical agent builder platforms.
00:09:07.519 --> 00:09:10.000
Then you have all the labs that are also offering products here.
00:09:10.159 --> 00:09:13.440
But I think something that makes open hands unique is that you're an open source product.
00:09:13.600 --> 00:09:18.559
So one, maybe what was the wedge into this market of being the all-encompassing platform?
00:09:18.639 --> 00:09:20.960
And then two, going the open source route.
00:09:21.360 --> 00:09:24.399
So I've been in commercial open source for for over a decade now.
00:09:24.480 --> 00:09:25.919
And it's just, it's kind of in my blood.
00:09:26.000 --> 00:09:27.440
I just I love building in the open.
00:09:27.679 --> 00:09:29.279
I love engaging a large community.
00:09:29.440 --> 00:09:35.840
Um, also from the commercial side, it is nice to just like it gives you a lot of distribution, it gives you a lot of mind share, and companies tend to trust it more, right?
00:09:36.080 --> 00:09:38.720
Especially for things like, you know, mission critical infrastructure.
00:09:38.960 --> 00:09:44.399
I think companies have been burned enough times building their infrastructure around something that's closed source that they don't control.
00:09:44.559 --> 00:09:52.399
And then, you know, the vendor in question ends up jacking up prices or they go out of business or something else happens and they're kind of left in the lurch.
00:09:52.559 --> 00:10:01.200
So I think a lot of companies we've spoken with, especially like some of the large, large banks that we've been working with, they've explicitly said they would not build around a closed source solution here.
00:10:01.279 --> 00:10:07.440
They are only interested in open source because they understand this is going to become the backbone of the developer experience at their organization.
00:10:07.600 --> 00:10:09.360
They need to know that they own this in the long term.
00:10:09.519 --> 00:10:21.039
They're willing to purchase extra commercial features that we can provide on top of it, but they need to know that the code that they're writing against open hands today is going to stay with them in the long term, whether or not they continue to engage in a relationship with us.
00:10:21.279 --> 00:10:21.600
Got it.
00:10:21.759 --> 00:10:25.360
Because a lot of open source companies projects are like bottoms-up adoption.
00:10:25.519 --> 00:10:27.759
Can you really evangelize the developer community?
00:10:27.919 --> 00:10:32.320
Is that something that you're focused in, or are you more focused on this top-down enterprise sale?
00:10:32.639 --> 00:10:34.240
It tends to be a combination of both.
00:10:34.639 --> 00:10:40.159
Typically, the way we go to market is that we do have a very like large open source community.
00:10:40.320 --> 00:10:48.399
We have a lot of folks who are, you know, downloading open hands, who are looking at our documentation, who are using our SDK to like build custom agents.
00:10:48.639 --> 00:10:56.080
And we can some of those folks like, you know, will sign up for our cloud service, some of them and we can see them inside of our docs, you know, by IP address.
00:10:56.159 --> 00:11:00.000
We can see, okay, we've got 12 folks from BREX who are all looking at our documentation.
00:11:00.080 --> 00:11:01.360
Like, let's go talk to BREX.
00:11:01.600 --> 00:11:10.720
And so once once we have like a bunch of engineers within an organization that are like clearly engaged with us in some way, they're starring our GitHub repo, they're opening issues, they're talking to us in Slack.
00:11:10.960 --> 00:11:18.320
Once we hit a certain threshold of that, that's when we'll start to reach out, do that more top-down motion and say, hey, you've got a bunch of ICs who are playing with this thing.
00:11:18.480 --> 00:11:32.720
You know, we'll start talking to a manager or a VP or a CTO and say, hey, we can help you put some arms around this, we can help you measure ROI, we can help you set budgets, and we can help you, you know, build a more scalable infrastructure rather than having a bunch of engineers running this thing on
their laptops.
00:11:32.960 --> 00:11:33.440
That makes sense.
00:11:33.519 --> 00:11:39.039
And I feel like a question I get a lot in my C is basically like how to build a developer community, how to hire a DevRel.
00:11:39.360 --> 00:11:43.759
Was there anything you tried that didn't work or anything that was really a sticking point for the company?
00:11:44.159 --> 00:11:45.120
It's a really tricky thing.
00:11:45.200 --> 00:11:48.159
I think developers are, you know, fairly allergic to marketing.
00:11:48.480 --> 00:11:51.440
They're a hard, a hard group to sell to, a hard group to make happy all the time.
00:11:51.600 --> 00:11:54.960
I think the biggest thing is just providing value with the product and in the open source.
00:11:55.039 --> 00:12:03.200
I think one of the easiest ways to, you know, kill an open source project, kill an open source community is to be aggressive about what you're holding back.
00:12:03.360 --> 00:12:14.000
And we've we've tried to have a really clearly articulated line between like what goes into the open source and what goes into the closed source in order to really help our community on like have a mental model for what they should expect.
00:12:14.159 --> 00:12:14.480
Right.
00:12:14.720 --> 00:12:21.200
The line that we take is that basically we want to make agentic technology available to every engineer in the world, right?
00:12:21.360 --> 00:12:23.519
We want to we want to democratize access to agents.
00:12:23.679 --> 00:12:29.840
So all of like our research team does, for instance, it goes into the open source and that becomes available to everyone.
00:12:29.919 --> 00:12:35.840
But it's really only good for an individual developer or maybe a small team running on their laptops, right?
00:12:36.000 --> 00:12:50.960
The second you want to scale into the cloud and start, you know, getting many developers collaborating together in the same instance of open hands, the second you want to start connecting into commercial tools like JIR and Bitbucket and GitLab and GitHub, then you need to start buying the commercial stuff.
00:12:51.279 --> 00:12:55.840
It's a pretty clear, like, okay, once you've got a team with the budget, it makes sense for you to start buying the commercial stuff.
00:12:55.919 --> 00:13:01.519
But if you are a hobbyist, uh there's really not a lot in the in the commercial product that you would get value from.
00:13:01.759 --> 00:13:01.919
Right.
00:13:02.000 --> 00:13:06.720
So it sounds like the goal isn't necessarily to monetize from the hobbyists or the individual developers.
00:13:07.039 --> 00:13:07.360
Exactly.
00:13:07.440 --> 00:13:12.080
We're looking for money from, you know, typically and organizations with like hundreds or thousands of engineers.
00:13:12.240 --> 00:13:19.200
If you're smaller than that, you fit very cleanly into the open source or like a free tier of our cloud stuff kind of bucket.
00:13:19.519 --> 00:13:19.840
Got it.
00:13:19.919 --> 00:13:25.600
And as the models get better, how do you see the open hands platform evolving or transforming?
00:13:25.759 --> 00:13:28.559
I assume this is mostly tailwinds for you all.
00:13:28.960 --> 00:13:32.240
It's really exciting because the models are getting better very quickly.
00:13:32.559 --> 00:13:38.240
I think Opus in particular has shifted the bar in terms of what's possible with agents.
00:13:38.559 --> 00:13:42.879
I think GPT-5.4 and GPT-5.5 also, you know, we're a close follow there.
00:13:43.039 --> 00:13:48.000
I think that the end state is, you know, often what I'm hearing folks call the dark factory, right?
00:13:48.159 --> 00:14:00.000
Where, you know, you essentially have agents doing the entire loop of software development where they're um they're writing code, uh, they're doing the QA and quality control over that code, doing code review, et cetera.
00:14:00.159 --> 00:14:10.639
They're merging it, they're shipping to production, they're watching the production logs, they're watching production traffic, making sure it's behaving appropriately, and then maybe making additional changes and tuning based on what they see in prod.
00:14:10.960 --> 00:14:14.799
And then you've basically got humans inputting into that, into that dark factory.
00:14:15.120 --> 00:14:25.679
One from the product side, I think you're you're always going to need humans with vision for like what it is that we want to build and talking to users and people who have empathy for the end user and a sense for like, what is the vision of this company?
00:14:25.759 --> 00:14:26.879
What is the vision of this product?
00:14:27.120 --> 00:14:32.240
Who are inputting into that dark factory and saying, you know, okay, agents, go and build this thing, go and make this new feature.
00:14:32.559 --> 00:14:43.600
I think you're also going to need human beings who are kind of systems experts who can look at this very chaotic mess of agents and help to tune it for cost, speed, efficiency, accuracy.
00:14:43.840 --> 00:14:49.759
I think there's there's always going to be a need for folks to kind of like manage that chaotic system.
00:14:50.240 --> 00:14:56.320
Sometimes you'll hear people say, you know, it can just be agents talking to the users and agents talking, agents doing the tuning of other agents.
00:14:56.639 --> 00:14:58.879
I don't really see that being uh being a real end state.
00:14:58.960 --> 00:15:02.720
I think there's always going to be people involved, but they're going to set out a higher level of abstraction.
00:15:02.960 --> 00:15:21.440
But I would say our goal with open hands as the models improve and as this technology gets better and better and as the human agent interfaces improve is to offload as much of the toil of software development into the agents and really let the human beings focus on those like creative and highly technical problems that are involved in like building great software.
00:15:21.759 --> 00:15:26.960
And when selling to enterprises that
aren't as convinced yet, what is typically the convincing factor?
00:15:27.120 --> 00:15:30.480
How do you think about sales, especially when you don't have a sales background?
00:15:30.799 --> 00:15:35.840
We've actually started dividing since we we we brought on a CRO at the end of last year, which has been really helpful.
00:15:35.919 --> 00:15:38.240
We were doing just all founder-led sales out of that.
00:15:38.320 --> 00:15:40.080
And it was there were some learnings involved in that.
00:15:41.039 --> 00:15:42.399
Some very long-running POCs.
00:15:42.480 --> 00:15:47.039
We had one POC going that was literally like a year-long working with them.
00:15:47.279 --> 00:15:52.799
And then within six weeks of the CRO landing, they closed the deal and like it's you know mid-six-figure deal.
00:15:52.879 --> 00:15:54.240
And like, but it was awesome.
00:15:54.480 --> 00:15:58.799
He's he's done a very good job of putting some discipline in place of like, this is how we run a POC.
00:15:58.879 --> 00:16:01.200
You know, we're gonna put some clear success criteria in place.
00:16:01.360 --> 00:16:02.799
It's not gonna be more than two months.
00:16:02.879 --> 00:16:04.559
And so that's been really, really helpful.
00:16:04.720 --> 00:16:10.559
And typically what we do is is we we define a clear use case for our technology.
00:16:10.799 --> 00:16:20.240
That might be code review, it might be dependency management, it might be resolving security vulnerabilities, but it's it's generally a use case that we've already vetted out and we've done successfully with other companies.
00:16:20.480 --> 00:16:24.960
Those three are like, you know, very, very typical menu options for us when running the POC.
00:16:25.120 --> 00:16:33.120
And then we'll just demonstrate to them, like, hey, we can do this for one repo and we'll we'll show clear value, we'll get the buy-in from the engineers, we get the technical win, and then we say, okay, now let's scale it up.
00:16:33.200 --> 00:16:38.080
Let's take that same use case and apply it across your entire fleet of of repos, across your entire engineering org.
00:16:38.159 --> 00:16:41.679
And then let's scale along the other dimension and add more use cases to the to the mix.
00:16:41.759 --> 00:16:48.399
I would say that kind of like land and expand, like show value with one use case in one team and then and then expand from there is a big, a big piece.
00:16:48.639 --> 00:16:51.360
And when you're going after that one use case, let's take code review.
00:16:51.519 --> 00:16:54.000
There's a ton of code review products out there.
00:16:54.159 --> 00:16:58.879
And so it's like how do you win in one vertical is a horizontal platform.
00:16:59.200 --> 00:17:08.000
I think this is still a really big open question for the industry is to what degree do you end up with horizontal uh solutions or vertical solutions, right?
00:17:08.400 --> 00:17:14.000
I think there is value in uh, for instance, we've we've said, hey, we're we're laser focused on code.
00:17:14.160 --> 00:17:16.079
We're not doing general agents, things like that.
00:17:16.160 --> 00:17:25.279
And we think that has worn out really well for us, that we like, we know exactly who we're selling to, we're selling to software engineers, we're not getting distracted by all this like back automation type stuff that is also possible with agents.
00:17:25.519 --> 00:17:29.519
Then there's companies that have gone, you know, even more vertical and said, hey, we just do code review.
00:17:29.680 --> 00:17:47.119
And you know, I I would argue that that is too narrow, that a lot of the same setup you're doing to get a code review agent to understand your code base and work really well, it's the exact same setup you need to do to get an implementation agent working really well, to get a QA agent working really well, to get a production ops agent working really well.
00:17:47.599 --> 00:18:00.319
To some degree, you don't want to invest too much into a narrow solution, especially proprietary solution, where you're then going to have to go and repeat a bunch of that work in a slightly different format for a different, you know, narrow vertical solution.
00:18:00.480 --> 00:18:06.160
It's very clear that code kind of like fits into one vertically oriented bucket.
00:18:06.559 --> 00:18:10.319
And I I'm increasingly seeing companies recognize that.
00:18:10.559 --> 00:18:24.720
And rather than going with a like, you know, vertical code, like code review solution in particular, they're like building agents that are more generic that use open standards to do code review so that they can then take that same work and apply it to implementation, apply it to QA.
00:18:24.880 --> 00:18:28.559
I feel like we're also seeing companies move into the token path across every industry right now.
00:18:28.720 --> 00:18:34.160
The closer you are to the token path, hopefully the easier it is to monetize and establish some sort of pricing model.
00:18:34.480 --> 00:18:37.839
You guys have obviously been in the token path just given what you're building.
00:18:38.160 --> 00:18:42.880
Have you had to adjust your pricing model for a token maxing society?
00:18:42.960 --> 00:18:43.119
Yeah.
00:18:43.359 --> 00:18:47.039
Or are you just able to continue on the initial pricing wave you have?
00:18:47.359 --> 00:18:50.799
We've actually taken the stance that we don't price per token.
00:18:51.119 --> 00:18:53.839
We tend to ship open hands on-prem.
00:18:54.000 --> 00:18:59.200
So, so folks will host us in their own cloud, their private cloud, or in their data center, and they bring their own model, right?
00:18:59.279 --> 00:19:07.920
So they can bring anthropic, they can use use anthropic through bedrock, they can, you know, bring an open source model like Kimi or NemoTron, something like that.
00:19:08.240 --> 00:19:11.119
They can basically they they take care of the LLM themselves.
00:19:11.359 --> 00:19:14.400
And then we're just
charging typical licensing fees for our software.
00:19:14.799 --> 00:19:16.319
That's a little bit of a double-edged sword.
00:19:16.400 --> 00:19:25.920
I think you see some companies reporting like these eye-popping revenue numbers, and like we do get a little bit of FOMO seeing those, seeing those numbers, but then you look at their margins and it's like they're running at negative gross margin.
00:19:26.000 --> 00:19:28.079
And it's like, that's that's not the kind of revenue that I want.
00:19:28.160 --> 00:19:36.559
It like it might look nice on paper or to an investor saying, hey, you jump from you know one million to a hundred million in a year, but like you're losing money on every token.
00:19:36.720 --> 00:19:38.960
That's not a healthy business in my in my book, right?
00:19:39.039 --> 00:19:45.839
So we've made the choice of really focusing on more software licensing revenue rather than than margin token revenue.
00:19:46.160 --> 00:19:55.200
Are you spending up RL environments for these different use cases and then running that back through the model or are you not at that point yet?
00:19:55.279 --> 00:19:58.480
Because what you're able to do everything with the current setup?
00:19:58.880 --> 00:20:00.400
Our research team is doing a bit of that.
00:20:00.559 --> 00:20:01.440
We're experimenting with it.
00:20:01.519 --> 00:20:04.960
I think especially around we're really interested in skills.
00:20:05.440 --> 00:20:15.359
So we're we're especially starting to look at so companies are investing in skills for, you know, code review or bug fixing, or like, you know, how to work with this particular repo.
00:20:15.519 --> 00:20:17.920
And we've now built a feedback loop.
00:20:18.079 --> 00:20:26.960
Uh so we have a system now where we can watch a skill in production over time and see, you know, okay, this this code review skill has triggered, you know, a thousand times within this organization.
00:20:27.119 --> 00:20:31.839
Uh we can see, you know, here are the times where it did the right thing and it's its comments got merged.
00:20:32.000 --> 00:20:34.559
Here are the times where it did the wrong thing and its comments got ignored.
00:20:34.720 --> 00:20:38.960
And then we can, you know, feed that, feed that back through the system to improve the skill over time.
00:20:39.200 --> 00:20:55.119
And so we're, we're, we're building that that pipeline essentially, where we can iteratively improve skills and iteratively improve an underlying open source model as well in order to, you know, make the make the agent better and better at those particular tasks, those like narrow, narrow tasks.
00:20:55.440 --> 00:20:57.839
Well, I mean, what do you think happens to the open source market?
00:20:58.000 --> 00:21:02.000
Because right now we don't necessarily have the go-to open source model that folks are using.
00:21:02.160 --> 00:21:07.200
The benchmarks alone show that the Chinese models are proliferating and ahead.
00:21:07.519 --> 00:21:09.200
Do you think that continues to be the case?
00:21:09.279 --> 00:21:18.960
Or do you think a company like Reflection or Nvidia really starts to pick up the pace in the US and like we shift towards this model and maybe more away from closed source models?
00:21:19.279 --> 00:21:23.119
I think there's there's a lot of speculation as to why the the Chinese models have done so well.
00:21:23.279 --> 00:21:33.920
There's Anthropic in particular has accused them of like distilling its models, which I think the the Western model builders are, you know, recognize that is probably a legal gray area they don't want to be playing in.
00:21:34.000 --> 00:21:39.359
And so there's a natural disadvantage, I think, for the Western models who are, you know, playing by the rules essentially.
00:21:39.519 --> 00:21:46.160
I do think we're going to need to find a way to make sure that the West is developing competitive open source models.
00:21:46.319 --> 00:21:49.759
I'm very excited to see what comes out of reflection and if they can build something competitive.
00:21:49.839 --> 00:21:51.839
I think NVIDIA is investing a lot here.
00:21:52.079 --> 00:21:56.799
I think they've essentially said, like, hey, we want to be the open source model company in the West.
00:21:57.119 --> 00:22:06.960
So I'm I'm really excited about the competitive landscape in the West and the fact that there are a bunch of really smart folks running at this problem and figuring how do we how do we train models from scratch and do it the right way.
00:22:07.279 --> 00:22:14.000
Every month at this point, we're making like massive strides with the autonomous software engineer and where the future of coding is going.
00:22:14.160 --> 00:22:17.119
But do you think we will end up in an agent-to-agent world?
00:22:17.200 --> 00:22:21.359
Or if we're sitting here in a few years, how does the space look different than where it looks today?
00:22:21.839 --> 00:22:24.640
Multi-agent is definitely increasingly becoming a thing.
00:22:24.799 --> 00:22:29.200
I think what's interesting is that they tend to interact through human systems.
00:22:29.440 --> 00:22:43.200
So, like the the most common agent, like multi-agent setup that I see today is I'll have one agent that I control doing the implementation of a particular feature or fixing a bug or something like that.
00:22:43.359 --> 00:22:47.200
And then another agent that my team controls does the code review on that.
00:22:47.440 --> 00:22:50.160
And like, you know, my team controls the prompts for the code review.
00:22:50.240 --> 00:22:52.240
I control the prompt that drives the implementation.
00:22:52.559 --> 00:22:54.400
But they're they're communicating through GitHub.
00:22:54.480 --> 00:22:56.079
They're communicating through GitHub comments, right?
00:22:56.160 --> 00:22:57.599
And so I can see their interactions there.
00:22:57.680 --> 00:23:02.559
It's not like there's A to A protocol where they're talking to each other under
the hood in a way that I can't see.
00:23:02.720 --> 00:23:10.720
I think that that seems to be really important for the like human-agent interface that like the agents are communicating through the same platforms that humans communicate through.
00:23:10.799 --> 00:23:12.720
It like fits our model of the world very well.
00:23:12.880 --> 00:23:22.799
I think that's going to be probably the dominant kind of like multi-agent pattern is that agents are talking to each other through Slack, they're talking to each other through GitHub, they're talking to each other through Jira or Linear.
00:23:23.119 --> 00:23:35.440
The place that I see something like A to A potentially coming up, if we can get to a world where there is where prompt injection is a lot harder, I think still prompt injection is difficult, is when you can get agents communicating across trust boundaries, right?
00:23:35.599 --> 00:23:45.839
If you know you can get, you know, an agent at Bank of America talking to an agent from American Express about, you know, some kind of transaction that needs to happen, but they can, they, you know, it's a much looser protocol.
00:23:45.920 --> 00:23:49.200
And you know, right now those two companies would have APIs that are talking to each other, right?
00:23:49.359 --> 00:23:56.480
But if you could have a much looser protocol where two agents are negotiating through some some more tricky process, uh, I think that could be really, really powerful.
00:23:56.640 --> 00:24:05.519
But right now, if there was that public kind of interface uh into a company through an agent, it's just too easy to trick that agent into doing something that the company wouldn't want it to do.
00:24:05.680 --> 00:24:10.160
So I think I think we're we're a ways away from having agents that communicate across trust boundaries.
00:24:10.319 --> 00:24:26.880
I spoke with one company a while ago that was specifically would have an agent basically watch your individual slacks from different companies and then you know hold all that information in the background, but then suggest partnerships when there was like a clear, yeah, a clear pattern of like, okay, these two companies like both have a need that the other one could solve.
00:24:27.200 --> 00:24:27.599
So smart.
00:24:27.759 --> 00:24:29.839
Well, I had a few quick fire questions for you.
00:24:30.319 --> 00:24:30.720
That's okay.
00:24:31.039 --> 00:24:31.440
Awesome.
00:24:31.759 --> 00:24:36.240
If you could hand an agent something in your own life today that's not work related, what would it be?
00:24:36.559 --> 00:24:45.279
I spend a lot of time just like doing tiny little chores on the internet, you know, filling out my credit card details and, you know, going to buy a thing on Amazon.
00:24:45.440 --> 00:24:49.359
Like I don't need to go hunt for the right set of double A batteries on Amazon, right?
00:24:49.599 --> 00:24:56.559
That's the kind of thing that like I would love to just say to an agent, hey, buy me some double A batteries, make sure they're here by Wednesday, right?
00:24:56.799 --> 00:25:06.880
I would love to say, you know, book me a flight, you know, those, those little, little chores, you know, buy me concert tickets for this day or this band, those little chores where I'm still squinting at my screen and typing in a bunch of details.
00:25:07.039 --> 00:25:09.839
And I would love to offload that type of stuff.
00:25:10.160 --> 00:25:10.640
Agreed.
00:25:10.880 --> 00:25:14.559
What is the single hardest unsolved problem in software agents right now?
00:25:14.880 --> 00:25:16.559
The biggest thing is verification, right?
00:25:16.640 --> 00:25:32.319
We're really good at generating lots of code, but we're stuck in code review and we're stuck in doing QA work, getting agents that code review is is increasingly solved well, but agents that can also run the application, you know, take a video of itself clicking around the web app and showing that it does the thing it said it was gonna do.
00:25:32.640 --> 00:25:38.640
That will be a huge unlock because then we can get agents, you know, basically closing the loop and merging their own code.
00:25:39.200 --> 00:25:42.000
Is there a unique contribution you've gotten from the community?
00:25:42.400 --> 00:25:44.559
Lots of unique contributions, I would say.
00:25:44.720 --> 00:25:49.279
Probably the biggest ones are around the agents themselves and how they behave.
00:25:49.599 --> 00:25:51.440
We've gotten some really great contributions.
00:25:51.599 --> 00:25:53.200
I'd say two, two major ones.
00:25:53.359 --> 00:25:55.359
One around how we manage context windows.
00:25:55.519 --> 00:26:01.119
There's been a lot of tuning of like how do you selectively forget information as the context window grows throughout the course of a session.
00:26:01.279 --> 00:26:03.279
And then two, around security analysis.
00:26:03.440 --> 00:26:09.279
So every time the agent makes a tool call, how do we decide if this is a high, medium, or low risk action for the agent to take?
00:26:09.519 --> 00:26:11.680
That's a really important thing for us to be able to measure.
00:26:11.759 --> 00:26:16.640
And the community has done some really amazing work on building a pluggable interface for security analysis.
00:26:17.119 --> 00:26:21.680
And last one, is there something you know now that you wish you knew at the beginning of open hands?
00:26:22.079 --> 00:26:25.680
I think I I wish I'd been prepared for how fast this space was going to move.
00:26:25.839 --> 00:26:28.720
It is changing literally on a month-to-month basis.
00:26:28.880 --> 00:26:30.319
And it's very, very exciting.
00:26:30.400 --> 00:26:33.279
It's a really exciting time to be building, but it's also a little exhausting.
00:26:33.440 --> 00:26:34.319
Yeah, I agreed.
00:26:34.559 --> 00:26:35.200
I'm tired.
00:26:35.599 --> 00:26:36.799
Can you imagine how you feel?
00:26:37.039 --> 00:26:37.519
Yeah, yeah.
00:26:37.680 --> 00:26:38.640
Well, thank you, Ralway.
00:26:38.799 --> 00:26:39.440
This was really fun.
00:26:39.519 --> 00:26:40.480
I appreciate you coming on.
00:26:40.640 --> 00:26:41.359
Yeah, thanks for having me.
00:26:41.440 --> 00:26:42.799
This is a fun conversation.