WEBVTT
00:00:00.160 --> 00:00:09.919
Welcome to the inaugural episode of Bench Talks, Snorkel AI's podcast on benchmarks, evaluation, data quality, and real-world impact.
00:00:10.160 --> 00:00:20.559
Today, Vincent Sun Chen sits down with Alex Shaw, co-creator of Terminal Bench and Harbor, and a founding member of technical staff at Laud Institute.
00:00:20.879 --> 00:00:26.079
Once you have tasks, in theory, like that becomes a verifier for your agent to start creating the automation itself.
00:00:26.239 --> 00:00:26.480
Right.
00:00:26.640 --> 00:00:35.520
So that gets deployed into production, you get production traces, you take those traces, you turn them into tasks, and like hopefully it's like this is like the flywheel or the loop everyone talks about.
00:00:35.679 --> 00:00:40.320
But I do think that um that is probably the North Star we're fielding towards.
00:00:49.039 --> 00:00:50.000
Thank you for having me.
00:00:50.240 --> 00:00:52.000
Yeah, we're super excited to have you here.
00:00:52.159 --> 00:00:57.439
So I guess first of all, four months ago, we talked a bit about TV2 Harbor.
00:00:57.920 --> 00:00:58.640
That's launched.
00:00:58.799 --> 00:01:00.320
It's had a ton of momentum since then.
00:01:00.479 --> 00:01:03.200
What has been most surprising in the last four months?
00:01:03.520 --> 00:01:04.959
It's a good question.
00:01:05.200 --> 00:01:08.480
I think, okay, two things have been pretty surprising.
00:01:08.640 --> 00:01:13.840
So first is how quickly the model developers have hill climbed General Lynch 2.
00:01:14.239 --> 00:01:20.239
So I think when we were making it, while it was in development, 20%, 30% is the scores that we were seeing.
00:01:20.400 --> 00:01:25.280
And then by the time we actually release it, like the better models were scoring, I think, up to 40 or 45%.
00:01:25.760 --> 00:01:29.760
And now the most recent submissions from OpenAI, I think were like 75%.
00:01:30.079 --> 00:01:34.959
And then if you like really build like a custom harness around it, people are have even gotten up to 80%.
00:01:35.359 --> 00:01:37.599
So I guess in school, that still is a B minus.
00:01:37.680 --> 00:01:38.640
So it's fair enough.
00:01:38.799 --> 00:01:47.680
And we do, we've actually seen every single task in Terminal Bench 2 has been solved at least one time by one agent model combination over the course of the benchmark.
00:01:47.760 --> 00:01:49.680
So in theory, that means 100% is possible.
00:01:49.760 --> 00:01:51.760
So there's still 20% room to go.
00:01:51.840 --> 00:01:55.680
But yeah, we're like actively working on Terminal Bench 3 because of this very reason.
00:01:55.760 --> 00:02:01.359
Like these benchmarks are very hill climbable these days, which is good for all the users of the agents, right?
00:02:01.439 --> 00:02:03.280
Like they get better agents because of it.
00:02:03.359 --> 00:02:04.640
But yeah, that was surprising.
00:02:04.719 --> 00:02:09.439
And then the other surprising thing is how quickly Harbor was adopted.
00:02:09.520 --> 00:02:11.360
We put it out there as like an experimental thing.
00:02:11.840 --> 00:02:12.719
People seem to want this.
00:02:12.800 --> 00:02:14.800
We'll put it out there and see if people pick it up.
00:02:14.960 --> 00:02:16.080
But yeah, people seem to like it.
00:02:16.400 --> 00:02:16.879
That's awesome.
00:02:17.039 --> 00:02:22.879
On the first point, what do you think that says about either the benchmark or the models that, you know, they're hill climbing so quickly?
00:02:23.039 --> 00:02:25.680
Any lessons or hot takes on that phenomenon?
00:02:26.080 --> 00:02:28.000
It's a little bit confusing to be honest.
00:02:28.159 --> 00:02:31.360
So the Terminal Bench 2 tasks are, I think, very high quality.
00:02:31.520 --> 00:02:31.680
Yeah.
00:02:31.840 --> 00:02:34.319
Terminal bench three will be higher quality even.
00:02:34.400 --> 00:02:35.759
So I'm excited about that.
00:02:36.000 --> 00:02:40.400
But I think it's a relatively difficult benchmark to reward hack.
00:02:40.479 --> 00:02:44.960
So I do think the models are actually getting better at those specific types of tasks.
00:02:45.199 --> 00:02:49.840
But when I use like Claude Code myself, I use Claude Code all the time when I'm coding.
00:02:49.919 --> 00:02:52.159
And I see a lot of shortcomings still.
00:02:52.400 --> 00:02:52.960
Do you know what I mean?
00:02:53.199 --> 00:02:57.360
80% on terminal bench, you'd think it would just be like solving every problem instantaneously or something.
00:02:57.599 --> 00:02:59.520
But it introduces a ton of complexity.
00:02:59.599 --> 00:03:05.759
It still I think has a relatively hard time building off of its work and building like robust and secure code.
00:03:05.919 --> 00:03:06.240
Yeah.
00:03:06.560 --> 00:03:16.159
And it has me thinking a lot like, how do we even code more and more of these shortcomings into tasks that exist in either Terminal Bench 3 or just any other benchmark people are building?
00:03:16.319 --> 00:03:19.520
I think maybe the lesson learned is coding is an extremely broad domain.
00:03:19.680 --> 00:03:21.599
89 tasks aren't going to cover all of it.
00:03:21.680 --> 00:03:24.159
So we need a thousand times more benchmarks that we have right now.
00:03:24.319 --> 00:03:24.560
Yeah.
00:03:24.719 --> 00:03:30.319
And that's a big part of our philosophy in launching open benchmarks and kind of working on benchmarks as a core initiative for us too.
00:03:30.400 --> 00:03:34.719
Is it's not just about evaluating these models, looking backward, and snapshotting progress.
00:03:34.800 --> 00:03:38.080
It's also just shaping and defining, hey, where are these models going?
00:03:38.159 --> 00:03:40.639
What are the real ways that they're used in practice?
00:03:40.960 --> 00:03:44.560
And actually articulating that well is actually really hard.
00:03:44.639 --> 00:03:52.000
And captureing, hey, what am I doing in my day-to-day that is not in these benchmarks ends up being quite important, in my view, for closing that gap.
00:03:52.240 --> 00:03:53.840
So very much you read there.
00:03:54.719 --> 00:04:03.759
At a high level, what do you think made Terminal Bench 2 stick and acquire the big splash that it did on all the kind of recent model cars with Claude and Codex?
00:04:04.080 --> 00:04:06.000
Even as of today, I think that there's a new one out there.
00:04:06.319 --> 00:04:07.599
What do you think made it sick?
00:04:08.080 --> 00:04:12.639
I guess I should preface it by it does seem like we got a little bit lucky.
00:04:12.879 --> 00:04:23.120
And but by luck, what I mean is I think there you need a few different things to have a candidate for a benchmark that becomes a standard in the industry.
00:04:23.279 --> 00:04:26.560
One is I think two that you have the most control over is it needs to be high quality.
00:04:26.639 --> 00:04:26.800
Yeah.
00:04:26.959 --> 00:04:33.040
By high quality, tasks that people care about and tasks that are not easily reward hackable.
00:04:33.120 --> 00:04:33.920
That that's one.
00:04:34.160 --> 00:04:36.319
Two, I think it needs to be easy to adopt.
00:04:36.399 --> 00:04:38.879
So this is something that we agonize over with terminal bench two.
00:04:39.040 --> 00:04:48.160
And one is like even with terminal bench two, which led way to harbor, was like, what is the fewest characters somebody could type into their computer to start running Terminal Bench 2?
00:04:48.240 --> 00:04:54.000
So it's like in theory, if you have Docker installed and running, you could run it in a single line of code without having anything else installed on your computer.
00:04:54.240 --> 00:04:56.160
So that that I think was a big one.
00:04:56.319 --> 00:05:08.560
And then the last one that you have maybe a little bit less control over is making a correct bet on a capability that people are going to care about that currently is not being measured by existing benchmarks.
00:05:08.720 --> 00:05:08.879
Right.
00:05:09.040 --> 00:05:17.040
So for us, this was the bet on the terminal as a powerful tool for computer use, not just coding, but like actually just doing arbitrary tasks on a computer.
00:05:17.199 --> 00:05:25.759
And I do have to credit Andy Konwinsky, the co-founder of Laud Institute, data version perplexity, and then Ludwig Schmidt, the Stanford professor.
00:05:25.839 --> 00:05:26.079
Yeah.
00:05:26.319 --> 00:05:29.839
For I think a lot of the initial ideas around terminal came from them.
00:05:30.000 --> 00:05:30.079
Yeah.
00:05:30.240 --> 00:05:33.199
And they were thinking, like, hey, it's a text-based tool that's extremely powerful.
00:05:33.360 --> 00:05:34.720
So I think they really nailed that.
00:05:34.800 --> 00:05:37.759
And I think Mike and I executed well on it and expanded the vision.
00:05:38.000 --> 00:05:40.480
I love that articulation and I agree on all fronts.
00:05:40.639 --> 00:05:44.560
To go a little deeper on the bed of the terminal, why do you think that's the right abstraction?
00:05:44.720 --> 00:05:45.439
Because I agree with you.
00:05:45.519 --> 00:05:49.600
It feels like it's not just a kind of CLI agent.
00:05:49.680 --> 00:05:51.600
It's way more general purpose than that.
00:05:51.680 --> 00:05:55.519
It's a kind of much broader affordance for general computer use and general purpose tasks.
00:05:55.759 --> 00:06:00.560
Why do you think the terminal is such a good abstraction or kind of core foundational layer to build on?
00:06:00.959 --> 00:06:09.759
I'll say what I've said since the beginning, which is before UIs even existed, it was just terminals, or it was the way that you interacted with a computer.
00:06:09.920 --> 00:06:19.120
So it makes sense that most things you do on your computer that you do through a UI originally maybe were either done through a terminal or can now be done through a terminal.
00:06:19.360 --> 00:06:28.399
So I think just in general, these are large language models and they tend to excel at text-based domains more than they do at vision-based domains.
00:06:28.560 --> 00:06:30.560
A lot of user interface is vision-based.
00:06:30.720 --> 00:06:36.480
So I think because of that, they excel at writing commands and scripts and code in the terminal.
00:06:36.639 --> 00:06:39.839
And then I think people underestimate how much you can do with the terminal.
00:06:40.000 --> 00:06:49.519
Cloud Code released Cloud Cowork is literally just like a UI wrap around Cloud Code for people to do like document style work, like lawyers and stuff like that.
00:06:49.680 --> 00:07:01.120
And then even myself, like when I, for example, when I like shop for groceries, I will go to this app called Pabrika and I'll pick a recipe and I'll add groceries to my grocery list, and then maybe I'll export them into my reminders app.
00:07:01.199 --> 00:07:03.279
And I'm doing this all like through my user interface.
00:07:03.519 --> 00:07:08.560
But just as an experiment the other day, I asked Cloud Code, like, hey, can you see my paprika app?
00:07:08.720 --> 00:07:11.600
And it was like, I found the SQLite database for it.
00:07:11.759 --> 00:07:12.800
And I was like, Can you read it?
00:07:12.879 --> 00:07:13.680
I'm seeing my recipes.
00:07:13.759 --> 00:07:15.680
And it's like, yeah, I just read your SQLite database.
00:07:15.759 --> 00:07:17.040
So I see all of your recipes.
00:07:17.120 --> 00:07:19.759
And I was like, can you pick a couple and then and then make a grocery list?
00:07:19.839 --> 00:07:20.800
I was like, yeah, okay, I did that.
00:07:20.879 --> 00:07:23.279
I was like, can you export that to the reminders app?
00:07:23.360 --> 00:07:32.240
And it like found some scripting library that Apple has created where you can like interact with Apple applications on your computer through the terminal and added it to my reminders app.
00:07:32.399 --> 00:07:39.120
So it's it's just to show that you can do, I think, really almost anything on a computer through through a terminal.
00:07:39.360 --> 00:07:39.600
Yeah.
00:07:39.680 --> 00:07:40.000
Yeah.
00:07:40.319 --> 00:07:41.199
I think that's well said.
00:07:41.360 --> 00:07:47.120
And I think more broadly on the quality point you mentioned, task quality as a differentiation for TB2.
00:07:47.439 --> 00:07:48.560
What do you think contributes to that?
00:07:48.720 --> 00:07:54.240
How do you maintain such high quality on a benchmark at the scale you were trying to run towards?
00:07:54.639 --> 00:07:55.199
Yeah.
00:07:55.519 --> 00:07:58.319
Originally, like there was no scale.
00:07:58.480 --> 00:08:06.000
Originally it was just terminal bench one, and it was me and Mike making tasks ourselves, trying to hit some amount of tasks to put a benchmark out.
00:08:06.240 --> 00:08:12.319
At what point we realized like Mike and I don't have enough collective knowledge to create a hundred really interesting tasks.
00:08:12.399 --> 00:08:15.600
We maybe hit 10 each, and then quickly we're like, it's hard to come up with more.
00:08:15.680 --> 00:08:20.720
So then it became a crowdsourcing, crowd, crowdsourced benchmark where we went and asked for contributions.
00:08:21.279 --> 00:08:34.639
I think one thing so first and foremost, the thing that helps most is just having smart and capable contributors who understand or who are able to recreate difficult problems that they've solved historically in their careers.
00:08:34.799 --> 00:08:42.000
So for us, that was like Nicholas Carlini joining early and contributing like tons of tasks to terminal bench because he had just solved so many difficult problems.
00:08:42.240 --> 00:08:46.960
So I think just like having really expert level contributors makes a big difference.
00:08:47.039 --> 00:08:50.799
And then maybe also providing them with tools to check their own work, right?
00:08:50.960 --> 00:08:51.279
Essentially.
00:08:51.679 --> 00:09:00.559
So we created like rubrics and some nice stuff where you could actually like have LL and judge look at your task and be like, hey, it's well specified, like your instructions are missing, like output schemas or something like that.
00:09:00.799 --> 00:09:10.480
And then the last check, or maybe the last step of the process, which was really Mike invented a lot of this, was the QA process, which is once we had the tasks, right?
00:09:10.639 --> 00:09:15.679
How are we now gonna just put them through a bunch of thorough testing to see like where do they break?
00:09:15.759 --> 00:09:16.399
Are they brittle?
00:09:16.559 --> 00:09:17.600
Are they reward hackable?
00:09:17.759 --> 00:09:19.200
Should they make it into the benchmark?
00:09:19.360 --> 00:09:23.440
And I think that all of those things combined led to a high quality benchmark.
00:09:23.600 --> 00:09:23.919
Yeah.
00:09:24.159 --> 00:09:25.519
I think that resonates a ton.
00:09:25.679 --> 00:09:31.039
Even as we're building data sets on our end, often what we're finding is generation isn't really the bottleneck.
00:09:31.120 --> 00:09:33.679
It's often the verification and quality control piece.
00:09:33.840 --> 00:09:46.000
And the more that you shift left in terms of making things more interactive or kind of giving feedback sooner, the more you have those types of checks or protocols in place, yeah, the more you can tighten that feedback loop for quality.
00:09:46.159 --> 00:09:48.559
So a lot of those principles resonate a ton.
00:09:48.720 --> 00:09:53.840
I know there's also even some kind of recent you guys have become so big that people are trying to cheat the dashboard.
00:09:53.919 --> 00:10:03.120
Uh what has it been like to manage kind of community there more broadly and really keep the integrity of the benchmark as this thing has blown up quite a bit?
00:10:03.519 --> 00:10:07.039
Yeah, I so we have like simple checks in place.
00:10:07.200 --> 00:10:12.559
We I guess initially we just had people email us their results and we would go manually put it on the leaderboard.
00:10:12.639 --> 00:10:13.840
That didn't scale very well.
00:10:14.080 --> 00:10:23.519
So we started, we started a hugging face repo where people could submit their results to the hugging face repo and they would automatically get like simple validations like did they run the tasks enough times?
00:10:23.679 --> 00:10:25.519
Did they use the correct configuration, stuff like that?
00:10:25.679 --> 00:10:25.759
Yeah.
00:10:25.919 --> 00:10:28.080
And then those would upload to the leaderboard.
00:10:28.399 --> 00:10:35.600
Those kind of help with like basic things, but then really just having everyone's submissions publicly available, right?
00:10:35.759 --> 00:10:40.720
Is I think what really creates the final layer of verification, which is anybody can go in.
00:10:40.799 --> 00:10:44.639
Like this leaderboard entry that recently got taken down, we didn't catch it.
00:10:44.720 --> 00:10:51.519
It was some other contributor dug in and figured out that they had bundled some gold trajectories into their agent binary.
00:10:51.679 --> 00:10:54.720
And that helps us create the most high integrity benchmark.
00:10:54.799 --> 00:11:00.159
But I will say there's no easy way for us to detect, for example, like was the model fine-tuned on our actual solutions or something.
00:11:00.320 --> 00:11:01.840
There's things that you just can't verify.
00:11:02.000 --> 00:11:05.759
So I think the only solution to that is like people will overfit to your benchmark over time.
00:11:05.919 --> 00:11:08.480
You need to make new ones fast.
00:11:08.559 --> 00:11:08.720
Yeah.
00:11:08.879 --> 00:11:20.720
And that's what we're trying to do with Harbor, especially is like, how can we build the benchmark factory, like the machine that other people can use to make their benchmarks, as opposed to just creating our own benchmarks one by one?
00:11:20.879 --> 00:11:24.000
Because we need, I think, like I said, way more benchmarks than we have now.
00:11:24.159 --> 00:11:27.120
And we need them to come out quickly so we're not overfitting to the current one.
00:11:27.279 --> 00:11:28.080
Let's talk about Harbor.
00:11:28.320 --> 00:11:30.480
It's for way more than just Terminal Bench now.
00:11:30.559 --> 00:11:33.919
I think Suite Bench and many other third-party benchmarks are on the registry.
00:11:34.080 --> 00:11:34.799
What's the vision?
00:11:34.879 --> 00:11:38.559
How has it evolved since you literally launched it kind of just a few months ago?
00:11:38.799 --> 00:11:39.200
Yeah.
00:11:39.360 --> 00:11:47.840
Like I said before, when we put it out, it was mostly experimental, I think, just to see if we had built a lot of interesting and good infrastructure for running Terminal Bench.
00:11:47.919 --> 00:11:51.360
And we had, like I said before, cared a lot about making it easy to use.
00:11:51.519 --> 00:11:56.799
What is the simplest way to specify a task on a computer that an agent should automate?
00:11:56.960 --> 00:11:59.919
What is the simplest way to start getting rollouts on it at scale?
00:12:00.080 --> 00:12:01.200
So we put out Harbor.
00:12:01.679 --> 00:12:08.799
From the beginning, the vision was always we want a lot of the popular benchmarks to be adapted into the framework because it's flexible enough to support that.
00:12:08.879 --> 00:12:10.799
And we want people to create their own benchmarks.
00:12:10.879 --> 00:12:14.720
So I think uh we have seen like 10 or 20 benchmarks.
00:12:14.879 --> 00:12:19.519
The one I saw this week was called Runebench, and it's literally like agents playing RuneScape.
00:12:19.679 --> 00:12:24.639
If you guys know High Game, it's like a World of Warcraft style game that was popular probably like 10 or 20 years ago.
00:12:24.720 --> 00:12:26.960
And so yeah, people are using it to build interesting benchmarks.
00:12:27.039 --> 00:12:30.879
We've adapted like 40 or 50 benchmarks into it, like Street Bench, like you mentioned.
00:12:31.039 --> 00:12:38.480
And then maybe the surprising thing has been, or maybe the thing we're trying to do most right now is expand outside of just coding as a primary use case.
00:12:38.639 --> 00:12:46.480
So part of this is because of how we've seen people use it, like to create finance tasks, law tasks, like all of these things in the same vein as cloud cowork, right?
00:12:46.559 --> 00:12:53.519
Being being an extension of cloud code, it's why shouldn't all of these other benchmarks or types of tasks also fit into the Harbor framework?
00:12:53.600 --> 00:12:59.120
So we're just actively developing like Harbor towards just general agent automation tasks.
00:12:59.360 --> 00:12:59.759
Yeah.
00:13:00.000 --> 00:13:06.320
What do you think that Harbor does well that was previously a gap for researchers, agent builders in general?
00:13:06.720 --> 00:13:16.399
I so shortly after we made Terminal Bench, I got wind that Guillermo Rausch, the founder of Vercel and also the inventor of Next.js, yeah, was a fan of it.
00:13:16.480 --> 00:13:18.879
So I like tried to hop out call with Guillermo to talk to him.
00:13:19.039 --> 00:13:28.559
And the thing I wanted to tell him was what while building Terminal Bench, the thing that I was trying to do the most was like give it the same level of usability and developer experience that like a Next.js has.
00:13:28.639 --> 00:13:32.240
Like all these web frameworks that are known for just being like extremely nice and usable.
00:13:32.399 --> 00:13:37.360
So I think that we obviously took that same philosophy and instilled it into Harbor.
00:13:37.679 --> 00:13:52.399
And then I think Tinker is another good example of like how getting an extremely simple interface that abstracts away 90% of the complexity, but still allows you to have 90% of the flexibility you need is a really powerful tool.
00:13:52.559 --> 00:13:54.159
And I think that's what Harbor did well.
00:13:54.240 --> 00:14:04.480
Is there was a lot of frameworks out there for spinning up containerized tasks, like measuring an agent on it, but nobody had really gotten it to be super simple, super usable, and extremely flexible.
00:14:04.559 --> 00:14:05.759
And I think that's what we did right.
00:14:06.000 --> 00:14:06.240
Yeah.
00:14:06.399 --> 00:14:11.679
Where is Harbor going and why should people building or interacting with agents be excited about that?
00:14:12.000 --> 00:14:13.519
Yeah, it's a good question.
00:14:13.759 --> 00:14:29.200
I think what we've seen is right now, okay, one model development is an extremely empirical process, which is why these benchmarks are so important, is because often the benchmark is how you make decisions about what you're doing during the training phase.
00:14:29.360 --> 00:14:33.200
And I think what we've seen is agent development is an equally empirical process.
00:14:33.360 --> 00:14:41.759
Pretty much anything that touches a model, which is more or less a black box, it's not like some lines of code that you can go read and then know ahead of time what's going to happen.
00:14:41.840 --> 00:14:42.080
Right.
00:14:42.320 --> 00:14:49.200
You need tools to be able to measure how your agent is performing and whether your changes are improving it or harming it.
00:14:49.360 --> 00:14:56.559
And yeah, I think what we've also seen is data right now, tasks move between a lot of hands.
00:14:56.720 --> 00:14:57.039
Yeah.
00:14:57.200 --> 00:14:59.759
And it's very helpful to have a common language.
00:15:00.000 --> 00:15:08.000
So tasks, in a way, are a way of specifying these are the capabilities that we care about, that we're trying to optimize towards, like it's your product roadmap, like it's your product spec.
00:15:08.240 --> 00:15:17.120
And having a unified format and even framework that people can use to get value out of the data in that format, I think helps solve a coordination problem in the industry.
00:15:17.440 --> 00:15:18.639
Makes a ton of sense.
00:15:18.799 --> 00:15:22.639
I guess moving forward as you're trying to predict, hey, how do you build out Harbor?
00:15:22.879 --> 00:15:25.279
How do you build out the next versions of Terminal Bench?
00:15:25.440 --> 00:15:27.600
What do you view as the invariants moving forward?
00:15:27.679 --> 00:15:37.840
What are the things that aren't changing that you feel like will be true for many years to come as you continue to shape the future of where, again, eval's agentic development in general is going?
00:15:38.159 --> 00:15:39.519
It's a good question.
00:15:39.840 --> 00:15:44.879
I think I think there's a lot of talk right now about continual learning and learning from online only.
00:15:45.360 --> 00:15:54.480
I really hope that is a problem that gets solved because it just seems like those are it's fairly obvious how that would benefit all of the users of agents.
00:15:54.639 --> 00:15:54.879
Right.
00:15:55.039 --> 00:16:13.440
But I do think that in the foreseeable future, offline is going to be extremely important, which is like how do you take these workloads that are happening in online settings and then encode them into these tasks and environments that can be used during offline training, offline agent development, these sorts of things.
00:16:13.679 --> 00:16:22.480
And I don't see any reason why you can't like directly take what you're seeing in the online setting and try to encode that into these offline tasks and environments.
00:16:22.720 --> 00:16:29.519
But I think that that will probably continue to be a core piece of agent and model development and into the next few years.
00:16:29.840 --> 00:16:48.480
I think maybe the trend that we're starting to see now that I think will continue is a lot of companies are going to start caring a lot about data and evaluations, and they're going to be trying to turn their workflows into tasks that they can then start automating with agents and models.
00:16:48.720 --> 00:16:55.600
So I just think the demand for this type of data and these this type of optimization is about to explode.
00:16:55.840 --> 00:16:56.080
Yeah.
00:16:56.320 --> 00:17:08.880
One of the things that we believe internally, very much so, is hey, by shaping the data, you're shaping your product specs and you're really defining where not just your own company, enterprise products are going, but also where the field is going.
00:17:09.039 --> 00:17:16.400
And we were joking earlier that probably the best way to get a capability into these models moving forward is to contribute a terminal bench test, right?
00:17:16.480 --> 00:17:20.240
Because that's the way that these frontier labs are going to be benchmarking.
00:17:20.400 --> 00:17:22.960
That's the way that progress in hill climbing is going to be defined.
00:17:23.039 --> 00:17:28.799
And so um, thinking of data as the way to chart the roadmap for the whole space is something that resonates a ton.
00:17:29.039 --> 00:17:30.559
With that in mind, where do you want things to go?
00:17:30.720 --> 00:17:34.720
What are some benchmarks that you're excited to see out there that maybe don't exist today?
00:17:35.440 --> 00:17:41.039
So the benchmark that I personally want to see the most is a it's like a meta benchmark.
00:17:41.119 --> 00:17:45.359
It's a benchmark that measures the ability of agents to write verifiers.
00:17:45.680 --> 00:17:52.079
So it's so I think that verification is often the hardest part of task creation.
00:17:52.240 --> 00:17:52.559
Yeah.
00:17:52.799 --> 00:18:07.839
And you can usually tell if a task is a good one because the instruction will be relatively concise and often the verifier maybe will be relatively concise as well, or it will lean heavily on like existing verification infrastructure that exists in the real world.
00:18:08.000 --> 00:18:12.319
And I think creating the verifier is often the hardest part of creating a task.
00:18:12.400 --> 00:18:12.799
Yeah.
00:18:12.960 --> 00:18:18.720
And there's so many ways to like seed environments and seed instructions, but it's hard to seed verifiers.
00:18:18.799 --> 00:18:24.720
So in theory, it's okay, I could take my agent observability rollouts that that gives me instructions, maybe even environments.
00:18:24.880 --> 00:18:30.480
If I could just automate that verification piece, then I could just start like generating tasks like crazy.
00:18:30.720 --> 00:18:33.920
But right now we don't have a way of knowing like how good our agents at creating verifiers.
00:18:34.000 --> 00:18:35.440
And it's difficult to measure that.
00:18:35.839 --> 00:18:43.599
I think maybe you'd have to like create a ton of different solutions and make sure that all those solutions still pass against the verifier or something.
00:18:43.680 --> 00:18:44.720
Like you need some, yeah.
00:18:45.039 --> 00:18:46.480
What do you think that interface looks like?
00:18:46.640 --> 00:18:53.440
It sounds like it'd be an expert or human plus some sort of agent or model actually helping build these tasks.
00:18:53.599 --> 00:18:57.440
What is your view of what that ideal loop looks like in practice?
00:18:58.240 --> 00:19:11.440
I think I think it would look roughly like what I was just referring to, which is you take you take workflows that are encoded into production systems somehow.
00:19:11.519 --> 00:19:15.680
So this could be like Sweet Ench with GitHub is a perfect example of this.
00:19:15.839 --> 00:19:24.640
It's like that's a unique example where it's a workflow encoded into GitHub PR history, but it also happens to come with a verifier in that case.
00:19:24.799 --> 00:19:25.839
But that's typically not the case.
00:19:25.920 --> 00:19:34.079
So agent observability traces, other platforms that people do their work in, using those to like seed the environments and the instructions.
00:19:34.160 --> 00:19:34.400
Yeah.
00:19:34.640 --> 00:19:41.519
And then synthesizing verifiers probably need a human in the loop to do like last mile, take those things over the finish line.
00:19:41.599 --> 00:19:42.799
So you actually have tasks.
00:19:43.039 --> 00:19:48.559
And then once you have tasks, in theory, like that becomes a verifier for your agent to start creating the automation itself.
00:19:48.720 --> 00:19:48.960
Right.
00:19:49.119 --> 00:19:50.720
And then that gets to put into production.
00:19:50.799 --> 00:19:53.759
You get production traces, you take those traces, you turn them into tasks.
00:19:53.839 --> 00:19:57.599
And hopefully it's like this is like the flywheel or the loop everyone talks about.
00:19:57.680 --> 00:20:00.400
But I do think that that is probably the North Star worth building towards.
00:20:00.640 --> 00:20:01.359
That's super exciting.
00:20:01.440 --> 00:20:05.839
To me, it's about making the way we program agents way more accessible in general.
00:20:06.000 --> 00:20:16.079
The more you can build verifiers with experts in the loop, of course, searing where the capabilities are going, the more you're able to shape these models in the ways that people actually want to use them or businesses want to use them.
00:20:16.160 --> 00:20:17.119
I think it's a great bet.
00:20:17.359 --> 00:20:18.480
Okay, last question.
00:20:18.720 --> 00:20:25.279
What do you feel is something that people don't tell you about building benchmarks that that is generally surprising?
00:20:25.680 --> 00:20:34.400
I think it's more fun to create a task than people realize, especially if it's a problem that you have solved historically that was hard for you.
00:20:34.640 --> 00:20:40.559
It's it's at least for me, it's a fun mental exercise to think, okay, what is the best way to frame this up in a way that's verifiable?
00:20:40.720 --> 00:20:42.640
And then I get to go see if the agents can solve it.
00:20:42.720 --> 00:20:44.240
So that's maybe an unexpected thing.
00:20:44.400 --> 00:20:50.319
And then the other unexpected thing is I think how painful it is to review tasks.
00:20:50.480 --> 00:21:00.079
It's extremely difficult to look at somebody else's task, try to load it into your brain, decide whether or not it's implemented well and what capabilities it's testing.
00:21:00.240 --> 00:21:09.039
So I think it's a lot more of a grind than people think to review the benchmark and put it out, but it's a lot more exciting than they expect to actually like create the tasks in the benchmark.
00:21:09.279 --> 00:21:12.720
Anything else you want to share more broadly that I didn't ask about?
00:21:13.039 --> 00:21:17.759
I guess the only other thing I'd mention is we're working on terminal bench three right now.
00:21:18.000 --> 00:21:23.279
And the thing we want to do most is this is what I was actually telling Vincent beforehand.
00:21:23.440 --> 00:21:28.480
Offline is with Terminal Bench 1, it was mostly we tried to create, and then we had to like pretty much beg people for tasks.
00:21:28.640 --> 00:21:31.680
And then we achieved some level of success based off of it.
00:21:31.920 --> 00:21:38.720
Terminal bench two is a little bit easier to get tasks for like other PhC students and stuff because it was becoming more popular for benchmark.
00:21:38.880 --> 00:21:43.519
But I think now we're in a position where Terminal Bench 3 can be.
00:21:44.160 --> 00:21:49.680
I think it's for a lot of people, it's their best opportunity to influence model development.
00:21:49.839 --> 00:22:04.799
And if we get enough buy-in from people who are great at making tasks that otherwise can go make a little mini benchmark themselves, but decide, okay, this should be part of terminal bench three, we can create like The best benchmark out there that should like dictate model development for the next year or something like that.
00:22:05.039 --> 00:22:14.720
So this means like trying to get the best people at the best companies to come and contribute a task of the most difficult thing that they've had to solve that they would be paid a lot of money to solve.
00:22:14.880 --> 00:22:20.799
So yeah, we're looking for contributors for Terminal Bench 3, and we would love to get as many people as possible involved in that process.
00:22:20.960 --> 00:22:21.359
Love it.
00:22:21.440 --> 00:22:26.559
So if you want to shape AI and where the field's going, Terminal Bench 3, contribute and you'll play a big role.
00:22:26.720 --> 00:22:27.119
Awesome.
00:22:27.200 --> 00:22:27.920
Thanks so much, Alex.
00:22:28.160 --> 00:22:29.440
Christian Bench Talks.
00:22:29.519 --> 00:22:29.759
That's right.
00:22:30.000 --> 00:22:31.440
We'll be excited to have you back soon.
00:22:31.519 --> 00:22:32.079
Thank you so much.
00:22:32.319 --> 00:22:33.039
Thanks.
00:00:00.160 --> 00:00:09.919
Welcome to the inaugural episode of Bench Talks, Snorkel AI's podcast on benchmarks, evaluation, data quality, and real-world impact.
00:00:10.160 --> 00:00:20.559
Today, Vincent Sun Chen sits down with Alex Shaw, co-creator of Terminal Bench and Harbor, and a founding member of technical staff at Laud Institute.
00:00:20.879 --> 00:00:26.079
Once you have tasks, in theory, like that becomes a verifier for your agent to start creating the automation itself.
00:00:26.239 --> 00:00:26.480
Right.
00:00:26.640 --> 00:00:35.520
So that gets deployed into production, you get production traces, you take those traces, you turn them into tasks, and like hopefully it's like this is like the flywheel or the loop everyone talks about.
00:00:35.679 --> 00:00:40.320
But I do think that um that is probably the North Star we're fielding towards.
00:00:49.039 --> 00:00:50.000
Thank you for having me.
00:00:50.240 --> 00:00:52.000
Yeah, we're super excited to have you here.
00:00:52.159 --> 00:00:57.439
So I guess first of all, four months ago, we talked a bit about TV2 Harbor.
00:00:57.920 --> 00:00:58.640
That's launched.
00:00:58.799 --> 00:01:00.320
It's had a ton of momentum since then.
00:01:00.479 --> 00:01:03.200
What has been most surprising in the last four months?
00:01:03.520 --> 00:01:04.959
It's a good question.
00:01:05.200 --> 00:01:08.480
I think, okay, two things have been pretty surprising.
00:01:08.640 --> 00:01:13.840
So first is how quickly the model developers have hill climbed General Lynch 2.
00:01:14.239 --> 00:01:20.239
So I think when we were making it, while it was in development, 20%, 30% is the scores that we were seeing.
00:01:20.400 --> 00:01:25.280
And then by the time we actually release it, like the better models were scoring, I think, up to 40 or 45%.
00:01:25.760 --> 00:01:29.760
And now the most recent submissions from OpenAI, I think were like 75%.
00:01:30.079 --> 00:01:34.959
And then if you like really build like a custom harness around it, people are have even gotten up to 80%.
00:01:35.359 --> 00:01:37.599
So I guess in school, that still is a B minus.
00:01:37.680 --> 00:01:38.640
So it's fair enough.
00:01:38.799 --> 00:01:47.680
And we do, we've actually seen every single task in Terminal Bench 2 has been solved at least one time by one agent model combination over the course of the benchmark.
00:01:47.760 --> 00:01:49.680
So in theory, that means 100% is possible.
00:01:49.760 --> 00:01:51.760
So there's still 20% room to go.
00:01:51.840 --> 00:01:55.680
But yeah, we're like actively working on Terminal Bench 3 because of this very reason.
00:01:55.760 --> 00:02:01.359
Like these benchmarks are very hill climbable these days, which is good for all the users of the agents, right?
00:02:01.439 --> 00:02:03.280
Like they get better agents because of it.
00:02:03.359 --> 00:02:04.640
But yeah, that was surprising.
00:02:04.719 --> 00:02:09.439
And then the other surprising thing is how quickly Harbor was adopted.
00:02:09.520 --> 00:02:11.360
We put it out there as like an experimental thing.
00:02:11.840 --> 00:02:12.719
People seem to want this.
00:02:12.800 --> 00:02:14.800
We'll put it out there and see if people pick it up.
00:02:14.960 --> 00:02:16.080
But yeah, people seem to like it.
00:02:16.400 --> 00:02:16.879
That's awesome.
00:02:17.039 --> 00:02:22.879
On the first point, what do you think that says about either the benchmark or the models that, you know, they're hill climbing so quickly?
00:02:23.039 --> 00:02:25.680
Any lessons or hot takes on that phenomenon?
00:02:26.080 --> 00:02:28.000
It's a little bit confusing to be honest.
00:02:28.159 --> 00:02:31.360
So the Terminal Bench 2 tasks are, I think, very high quality.
00:02:31.520 --> 00:02:31.680
Yeah.
00:02:31.840 --> 00:02:34.319
Terminal bench three will be higher quality even.
00:02:34.400 --> 00:02:35.759
So I'm excited about that.
00:02:36.000 --> 00:02:40.400
But I think it's a relatively difficult benchmark to reward hack.
00:02:40.479 --> 00:02:44.960
So I do think the models are actually getting better at those specific types of tasks.
00:02:45.199 --> 00:02:49.840
But when I use like Claude Code myself, I use Claude Code all the time when I'm coding.
00:02:49.919 --> 00:02:52.159
And I see a lot of shortcomings still.
00:02:52.400 --> 00:02:52.960
Do you know what I mean?
00:02:53.199 --> 00:02:57.360
80% on terminal bench, you'd think it would just be like solving every problem instantaneously or something.
00:02:57.599 --> 00:02:59.520
But it introduces a ton of complexity.
00:02:59.599 --> 00:03:05.759
It still I think has a relatively hard time building off of its work and building like robust and secure code.
00:03:05.919 --> 00:03:06.240
Yeah.
00:03:06.560 --> 00:03:16.159
And it has me thinking a lot like, how do we even code more and more of these shortcomings into tasks that exist in either Terminal Bench 3 or just any other benchmark people are building?
00:03:16.319 --> 00:03:19.520
I think maybe the lesson learned is coding is an extremely broad domain.
00:03:19.680 --> 00:03:21.599
89 tasks aren't going to cover all of it.
00:03:21.680 --> 00:03:24.159
So we need a thousand times more benchmarks that we have right now.
00:03:24.319 --> 00:03:24.560
Yeah.
00:03:24.719 --> 00:03:30.319
And that's a big part of our philosophy in launching open benchmarks and kind of working on benchmarks as a core initiative for us too.
00:03:30.400 --> 00:03:34.719
Is it's not just about evaluating these models, looking backward, and snapshotting progress.
00:03:34.800 --> 00:03:38.080
It's also just shaping and defining, hey, where are these models going?
00:03:38.159 --> 00:03:40.639
What are the real ways that they're used in practice?
00:03:40.960 --> 00:03:44.560
And actually articulating that well is actually really hard.
00:03:44.639 --> 00:03:52.000
And captureing, hey, what am I doing in my day-to-day that is not in these benchmarks ends up being quite important, in my view, for closing that gap.
00:03:52.240 --> 00:03:53.840
So very much you read there.
00:03:54.719 --> 00:04:03.759
At a high level, what do you think made Terminal Bench 2 stick and acquire the big splash that it did on all the kind of recent model cars with Claude and Codex?
00:04:04.080 --> 00:04:06.000
Even as of today, I think that there's a new one out there.
00:04:06.319 --> 00:04:07.599
What do you think made it sick?
00:04:08.080 --> 00:04:12.639
I guess I should preface it by it does seem like we got a little bit lucky.
00:04:12.879 --> 00:04:23.120
And but by luck, what I mean is I think there you need a few different things to have a candidate for a benchmark that becomes a standard in the industry.
00:04:23.279 --> 00:04:26.560
One is I think two that you have the most control over is it needs to be high quality.
00:04:26.639 --> 00:04:26.800
Yeah.
00:04:26.959 --> 00:04:33.040
By high quality, tasks that people care about and tasks that are not easily reward hackable.
00:04:33.120 --> 00:04:33.920
That that's one.
00:04:34.160 --> 00:04:36.319
Two, I think it needs to be easy to adopt.
00:04:36.399 --> 00:04:38.879
So this is something that we agonize over with terminal bench two.
00:04:39.040 --> 00:04:48.160
And one is like even with terminal bench two, which led way to harbor, was like, what is the fewest characters somebody could type into their computer to start running Terminal Bench 2?
00:04:48.240 --> 00:04:54.000
So it's like in theory, if you have Docker installed and running, you could run it in a single line of code without having anything else installed on your computer.
00:04:54.240 --> 00:04:56.160
So that that I think was a big one.
00:04:56.319 --> 00:05:08.560
And then the last one that you have maybe a little bit less control over is making a correct bet on a capability that people are going to care about that currently is not being measured by existing benchmarks.
00:05:08.720 --> 00:05:08.879
Right.
00:05:09.040 --> 00:05:17.040
So for us, this was the bet on the terminal as a powerful tool for computer use, not just coding, but like actually just doing arbitrary tasks on a computer.
00:05:17.199 --> 00:05:25.759
And I do have to credit Andy Konwinsky, the co-founder of Laud Institute, data version perplexity, and then Ludwig Schmidt, the Stanford professor.
00:05:25.839 --> 00:05:26.079
Yeah.
00:05:26.319 --> 00:05:29.839
For I think a lot of the initial ideas around terminal came from them.
00:05:30.000 --> 00:05:30.079
Yeah.
00:05:30.240 --> 00:05:33.199
And they were thinking, like, hey, it's a text-based tool that's extremely powerful.
00:05:33.360 --> 00:05:34.720
So I think they really nailed that.
00:05:34.800 --> 00:05:37.759
And I think Mike and I executed well on it and expanded the vision.
00:05:38.000 --> 00:05:40.480
I love that articulation and I agree on all fronts.
00:05:40.639 --> 00:05:44.560
To go a little deeper on the bed of the terminal, why do you think that's the right abstraction?
00:05:44.720 --> 00:05:45.439
Because I agree with you.
00:05:45.519 --> 00:05:49.600
It feels like it's not just a kind of CLI agent.
00:05:49.680 --> 00:05:51.600
It's way more general purpose than that.
00:05:51.680 --> 00:05:55.519
It's a kind of much broader affordance for general computer use and general purpose tasks.
00:05:55.759 --> 00:06:00.560
Why do you think the terminal is such a good abstraction or kind of core foundational layer to build on?
00:06:00.959 --> 00:06:09.759
I'll say what I've said since the beginning, which is before UIs even existed, it was just terminals, or it was the way that you interacted with a computer.
00:06:09.920 --> 00:06:19.120
So it makes sense that most things you do on your computer that you do through a UI originally maybe were either done through a terminal or can now be done through a terminal.
00:06:19.360 --> 00:06:28.399
So I think just in general, these are large language models and they tend to excel at text-based domains more than they do at vision-based domains.
00:06:28.560 --> 00:06:30.560
A lot of user interface is vision-based.
00:06:30.720 --> 00:06:36.480
So I think because of that, they excel at writing commands and scripts and code in the terminal.
00:06:36.639 --> 00:06:39.839
And then I think people underestimate how much you can do with the terminal.
00:06:40.000 --> 00:06:49.519
Cloud Code released Cloud Cowork is literally just like a UI wrap around Cloud Code for people to do like document style work, like lawyers and stuff like that.
00:06:49.680 --> 00:07:01.120
And then even myself, like when I, for example, when I like shop for groceries, I will go to this app called Pabrika and I'll pick a recipe and I'll add groceries to my grocery list, and then maybe I'll export them into my reminders app.
00:07:01.199 --> 00:07:03.279
And I'm doing this all like through my user interface.
00:07:03.519 --> 00:07:08.560
But just as an experiment the other day, I asked Cloud Code, like, hey, can you see my paprika app?
00:07:08.720 --> 00:07:11.600
And it was like, I found the SQLite database for it.
00:07:11.759 --> 00:07:12.800
And I was like, Can you read it?
00:07:12.879 --> 00:07:13.680
I'm seeing my recipes.
00:07:13.759 --> 00:07:15.680
And it's like, yeah, I just read your SQLite database.
00:07:15.759 --> 00:07:17.040
So I see all of your recipes.
00:07:17.120 --> 00:07:19.759
And I was like, can you pick a couple and then and then make a grocery list?
00:07:19.839 --> 00:07:20.800
I was like, yeah, okay, I did that.
00:07:20.879 --> 00:07:23.279
I was like, can you export that to the reminders app?
00:07:23.360 --> 00:07:32.240
And it like found some scripting library that Apple has created where you can like interact with Apple applications on your computer through the terminal and added it to my reminders app.
00:07:32.399 --> 00:07:39.120
So it's it's just to show that you can do, I think, really almost anything on a computer through through a terminal.
00:07:39.360 --> 00:07:39.600
Yeah.
00:07:39.680 --> 00:07:40.000
Yeah.
00:07:40.319 --> 00:07:41.199
I think that's well said.
00:07:41.360 --> 00:07:47.120
And I think more broadly on the quality point you mentioned, task quality as a differentiation for TB2.
00:07:47.439 --> 00:07:48.560
What do you think contributes to that?
00:07:48.720 --> 00:07:54.240
How do you maintain such high quality on a benchmark at the scale you were trying to run towards?
00:07:54.639 --> 00:07:55.199
Yeah.
00:07:55.519 --> 00:07:58.319
Originally, like there was no scale.
00:07:58.480 --> 00:08:06.000
Originally it was just terminal bench one, and it was me and Mike making tasks ourselves, trying to hit some amount of tasks to put a benchmark out.
00:08:06.240 --> 00:08:12.319
At what point we realized like Mike and I don't have enough collective knowledge to create a hundred really interesting tasks.
00:08:12.399 --> 00:08:15.600
We maybe hit 10 each, and then quickly we're like, it's hard to come up with more.
00:08:15.680 --> 00:08:20.720
So then it became a crowdsourcing, crowd, crowdsourced benchmark where we went and asked for contributions.
00:08:21.279 --> 00:08:34.639
I think one thing so first and foremost, the thing that helps most is just having smart and capable contributors who understand or who are able to recreate difficult problems that they've solved historically in their careers.
00:08:34.799 --> 00:08:42.000
So for us, that was like Nicholas Carlini joining early and contributing like tons of tasks to terminal bench because he had just solved so many difficult problems.
00:08:42.240 --> 00:08:46.960
So I think just like having really expert level contributors makes a big difference.
00:08:47.039 --> 00:08:50.799
And then maybe also providing them with tools to check their own work, right?
00:08:50.960 --> 00:08:51.279
Essentially.
00:08:51.679 --> 00:09:00.559
So we created like rubrics and some nice stuff where you could actually like have LL and judge look at your task and be like, hey, it's well specified, like your instructions are missing, like output schemas or something like that.
00:09:00.799 --> 00:09:10.480
And then the last check, or maybe the last step of the process, which was really Mike invented a lot of this, was the QA process, which is once we had the tasks, right?
00:09:10.639 --> 00:09:15.679
How are we now gonna just put them through a bunch of thorough testing to see like where do they break?
00:09:15.759 --> 00:09:16.399
Are they brittle?
00:09:16.559 --> 00:09:17.600
Are they reward hackable?
00:09:17.759 --> 00:09:19.200
Should they make it into the benchmark?
00:09:19.360 --> 00:09:23.440
And I think that all of those things combined led to a high quality benchmark.
00:09:23.600 --> 00:09:23.919
Yeah.
00:09:24.159 --> 00:09:25.519
I think that resonates a ton.
00:09:25.679 --> 00:09:31.039
Even as we're building data sets on our end, often what we're finding is generation isn't really the bottleneck.
00:09:31.120 --> 00:09:33.679
It's often the verification and quality control piece.
00:09:33.840 --> 00:09:46.000
And the more that you shift left in terms of making things more interactive or kind of giving feedback sooner, the more you have those types of checks or protocols in place, yeah, the more you can tighten that feedback loop for quality.
00:09:46.159 --> 00:09:48.559
So a lot of those principles resonate a ton.
00:09:48.720 --> 00:09:53.840
I know there's also even some kind of recent you guys have become so big that people are trying to cheat the dashboard.
00:09:53.919 --> 00:10:03.120
Uh what has it been like to manage kind of community there more broadly and really keep the integrity of the benchmark as this thing has blown up quite a bit?
00:10:03.519 --> 00:10:07.039
Yeah, I so we have like simple checks in place.
00:10:07.200 --> 00:10:12.559
We I guess initially we just had people email us their results and we would go manually put it on the leaderboard.
00:10:12.639 --> 00:10:13.840
That didn't scale very well.
00:10:14.080 --> 00:10:23.519
So we started, we started a hugging face repo where people could submit their results to the hugging face repo and they would automatically get like simple validations like did they run the tasks enough times?
00:10:23.679 --> 00:10:25.519
Did they use the correct configuration, stuff like that?
00:10:25.679 --> 00:10:25.759
Yeah.
00:10:25.919 --> 00:10:28.080
And then those would upload to the leaderboard.
00:10:28.399 --> 00:10:35.600
Those kind of help with like basic things, but then really just having everyone's submissions publicly available, right?
00:10:35.759 --> 00:10:40.720
Is I think what really creates the final layer of verification, which is anybody can go in.
00:10:40.799 --> 00:10:44.639
Like this leaderboard entry that recently got taken down, we didn't catch it.
00:10:44.720 --> 00:10:51.519
It was some other contributor dug in and figured out that they had bundled some gold trajectories into their agent binary.
00:10:51.679 --> 00:10:54.720
And that helps us create the most high integrity benchmark.
00:10:54.799 --> 00:11:00.159
But I will say there's no easy way for us to detect, for example, like was the model fine-tuned on our actual solutions or something.
00:11:00.320 --> 00:11:01.840
There's things that you just can't verify.
00:11:02.000 --> 00:11:05.759
So I think the only solution to that is like people will overfit to your benchmark over time.
00:11:05.919 --> 00:11:08.480
You need to make new ones fast.
00:11:08.559 --> 00:11:08.720
Yeah.
00:11:08.879 --> 00:11:20.720
And that's what we're trying to do with Harbor, especially is like, how can we build the benchmark factory, like the machine that other people can use to make their benchmarks, as opposed to just creating our own benchmarks one by one?
00:11:20.879 --> 00:11:24.000
Because we need, I think, like I said, way more benchmarks than we have now.
00:11:24.159 --> 00:11:27.120
And we need them to come out quickly so we're not overfitting to the current one.
00:11:27.279 --> 00:11:28.080
Let's talk about Harbor.
00:11:28.320 --> 00:11:30.480
It's for way more than just Terminal Bench now.
00:11:30.559 --> 00:11:33.919
I think Suite Bench and many other third-party benchmarks are on the registry.
00:11:34.080 --> 00:11:34.799
What's the vision?
00:11:34.879 --> 00:11:38.559
How has it evolved since you literally launched it kind of just a few months ago?
00:11:38.799 --> 00:11:39.200
Yeah.
00:11:39.360 --> 00:11:47.840
Like I said before, when we put it out, it was mostly experimental, I think, just to see if we had built a lot of interesting and good infrastructure for running Terminal Bench.
00:11:47.919 --> 00:11:51.360
And we had, like I said before, cared a lot about making it easy to use.
00:11:51.519 --> 00:11:56.799
What is the simplest way to specify a task on a computer that an agent should automate?
00:11:56.960 --> 00:11:59.919
What is the simplest way to start getting rollouts on it at scale?
00:12:00.080 --> 00:12:01.200
So we put out Harbor.
00:12:01.679 --> 00:12:08.799
From the beginning, the vision was always we want a lot of the popular benchmarks to be adapted into the framework because it's flexible enough to support that.
00:12:08.879 --> 00:12:10.799
And we want people to create their own benchmarks.
00:12:10.879 --> 00:12:14.720
So I think uh we have seen like 10 or 20 benchmarks.
00:12:14.879 --> 00:12:19.519
The one I saw this week was called Runebench, and it's literally like agents playing RuneScape.
00:12:19.679 --> 00:12:24.639
If you guys know High Game, it's like a World of Warcraft style game that was popular probably like 10 or 20 years ago.
00:12:24.720 --> 00:12:26.960
And so yeah, people are using it to build interesting benchmarks.
00:12:27.039 --> 00:12:30.879
We've adapted like 40 or 50 benchmarks into it, like Street Bench, like you mentioned.
00:12:31.039 --> 00:12:38.480
And then maybe the surprising thing has been, or maybe the thing we're trying to do most right now is expand outside of just coding as a primary use case.
00:12:38.639 --> 00:12:46.480
So part of this is because of how we've seen people use it, like to create finance tasks, law tasks, like all of these things in the same vein as cloud cowork, right?
00:12:46.559 --> 00:12:53.519
Being being an extension of cloud code, it's why shouldn't all of these other benchmarks or types of tasks also fit into the Harbor framework?
00:12:53.600 --> 00:12:59.120
So we're just actively developing like Harbor towards just general agent automation tasks.
00:12:59.360 --> 00:12:59.759
Yeah.
00:13:00.000 --> 00:13:06.320
What do you think that Harbor does well that was previously a gap for researchers, agent builders in general?
00:13:06.720 --> 00:13:16.399
I so shortly after we made Terminal Bench, I got wind that Guillermo Rausch, the founder of Vercel and also the inventor of Next.js, yeah, was a fan of it.
00:13:16.480 --> 00:13:18.879
So I like tried to hop out call with Guillermo to talk to him.
00:13:19.039 --> 00:13:28.559
And the thing I wanted to tell him was what while building Terminal Bench, the thing that I was trying to do the most was like give it the same level of usability and developer experience that like a Next.js has.
00:13:28.639 --> 00:13:32.240
Like all these web frameworks that are known for just being like extremely nice and usable.
00:13:32.399 --> 00:13:37.360
So I think that we obviously took that same philosophy and instilled it into Harbor.
00:13:37.679 --> 00:13:52.399
And then I think Tinker is another good example of like how getting an extremely simple interface that abstracts away 90% of the complexity, but still allows you to have 90% of the flexibility you need is a really powerful tool.
00:13:52.559 --> 00:13:54.159
And I think that's what Harbor did well.
00:13:54.240 --> 00:14:04.480
Is there was a lot of frameworks out there for spinning up containerized tasks, like measuring an agent on it, but nobody had really gotten it to be super simple, super usable, and extremely flexible.
00:14:04.559 --> 00:14:05.759
And I think that's what we did right.
00:14:06.000 --> 00:14:06.240
Yeah.
00:14:06.399 --> 00:14:11.679
Where is Harbor going and why should people building or interacting with agents be excited about that?
00:14:12.000 --> 00:14:13.519
Yeah, it's a good question.
00:14:13.759 --> 00:14:29.200
I think what we've seen is right now, okay, one model development is an extremely empirical process, which is why these benchmarks are so important, is because often the benchmark is how you make decisions about what you're doing during the training phase.
00:14:29.360 --> 00:14:33.200
And I think what we've seen is agent development is an equally empirical process.
00:14:33.360 --> 00:14:41.759
Pretty much anything that touches a model, which is more or less a black box, it's not like some lines of code that you can go read and then know ahead of time what's going to happen.
00:14:41.840 --> 00:14:42.080
Right.
00:14:42.320 --> 00:14:49.200
You need tools to be able to measure how your agent is performing and whether your changes are improving it or harming it.
00:14:49.360 --> 00:14:56.559
And yeah, I think what we've also seen is data right now, tasks move between a lot of hands.
00:14:56.720 --> 00:14:57.039
Yeah.
00:14:57.200 --> 00:14:59.759
And it's very helpful to have a common language.
00:15:00.000 --> 00:15:08.000
So tasks, in a way, are a way of specifying these are the capabilities that we care about, that we're trying to optimize towards, like it's your product roadmap, like it's your product spec.
00:15:08.240 --> 00:15:17.120
And having a unified format and even framework that people can use to get value out of the data in that format, I think helps solve a coordination problem in the industry.
00:15:17.440 --> 00:15:18.639
Makes a ton of sense.
00:15:18.799 --> 00:15:22.639
I guess moving forward as you're trying to predict, hey, how do you build out Harbor?
00:15:22.879 --> 00:15:25.279
How do you build out the next versions of Terminal Bench?
00:15:25.440 --> 00:15:27.600
What do you view as the invariants moving forward?
00:15:27.679 --> 00:15:37.840
What are the things that aren't changing that you feel like will be true for many years to come as you continue to shape the future of where, again, eval's agentic development in general is going?
00:15:38.159 --> 00:15:39.519
It's a good question.
00:15:39.840 --> 00:15:44.879
I think I think there's a lot of talk right now about continual learning and learning from online only.
00:15:45.360 --> 00:15:54.480
I really hope that is a problem that gets solved because it just seems like those are it's fairly obvious how that would benefit all of the users of agents.
00:15:54.639 --> 00:15:54.879
Right.
00:15:55.039 --> 00:16:13.440
But I do think that in the foreseeable future, offline is going to be extremely important, which is like how do you take these workloads that are happening in online settings and then encode them into these tasks and environments that can be used during offline training, offline agent development, these sorts of things.
00:16:13.679 --> 00:16:22.480
And I don't see any reason why you can't like directly take what you're seeing in the online setting and try to encode that into these offline tasks and environments.
00:16:22.720 --> 00:16:29.519
But I think that that will probably continue to be a core piece of agent and model development and into the next few years.
00:16:29.840 --> 00:16:48.480
I think maybe the trend that we're starting to see now that I think will continue is a lot of companies are going to start caring a lot about data and evaluations, and they're going to be trying to turn their workflows into tasks that they can then start automating with agents and models.
00:16:48.720 --> 00:16:55.600
So I just think the demand for this type of data and these this type of optimization is about to explode.
00:16:55.840 --> 00:16:56.080
Yeah.
00:16:56.320 --> 00:17:08.880
One of the things that we believe internally, very much so, is hey, by shaping the data, you're shaping your product specs and you're really defining where not just your own company, enterprise products are going, but also where the field is going.
00:17:09.039 --> 00:17:16.400
And we were joking earlier that probably the best way to get a capability into these models moving forward is to contribute a terminal bench test, right?
00:17:16.480 --> 00:17:20.240
Because that's the way that these frontier labs are going to be benchmarking.
00:17:20.400 --> 00:17:22.960
That's the way that progress in hill climbing is going to be defined.
00:17:23.039 --> 00:17:28.799
And so um, thinking of data as the way to chart the roadmap for the whole space is something that resonates a ton.
00:17:29.039 --> 00:17:30.559
With that in mind, where do you want things to go?
00:17:30.720 --> 00:17:34.720
What are some benchmarks that you're excited to see out there that maybe don't exist today?
00:17:35.440 --> 00:17:41.039
So the benchmark that I personally want to see the most is a it's like a meta benchmark.
00:17:41.119 --> 00:17:45.359
It's a benchmark that measures the ability of agents to write verifiers.
00:17:45.680 --> 00:17:52.079
So it's so I think that verification is often the hardest part of task creation.
00:17:52.240 --> 00:17:52.559
Yeah.
00:17:52.799 --> 00:18:07.839
And you can usually tell if a task is a good one because the instruction will be relatively concise and often the verifier maybe will be relatively concise as well, or it will lean heavily on like existing verification infrastructure that exists in the real world.
00:18:08.000 --> 00:18:12.319
And I think creating the verifier is often the hardest part of creating a task.
00:18:12.400 --> 00:18:12.799
Yeah.
00:18:12.960 --> 00:18:18.720
And there's so many ways to like seed environments and seed instructions, but it's hard to seed verifiers.
00:18:18.799 --> 00:18:24.720
So in theory, it's okay, I could take my agent observability rollouts that that gives me instructions, maybe even environments.
00:18:24.880 --> 00:18:30.480
If I could just automate that verification piece, then I could just start like generating tasks like crazy.
00:18:30.720 --> 00:18:33.920
But right now we don't have a way of knowing like how good our agents at creating verifiers.
00:18:34.000 --> 00:18:35.440
And it's difficult to measure that.
00:18:35.839 --> 00:18:43.599
I think maybe you'd have to like create a ton of different solutions and make sure that all those solutions still pass against the verifier or something.
00:18:43.680 --> 00:18:44.720
Like you need some, yeah.
00:18:45.039 --> 00:18:46.480
What do you think that interface looks like?
00:18:46.640 --> 00:18:53.440
It sounds like it'd be an expert or human plus some sort of agent or model actually helping build these tasks.
00:18:53.599 --> 00:18:57.440
What is your view of what that ideal loop looks like in practice?
00:18:58.240 --> 00:19:11.440
I think I think it would look roughly like what I was just referring to, which is you take you take workflows that are encoded into production systems somehow.
00:19:11.519 --> 00:19:15.680
So this could be like Sweet Ench with GitHub is a perfect example of this.
00:19:15.839 --> 00:19:24.640
It's like that's a unique example where it's a workflow encoded into GitHub PR history, but it also happens to come with a verifier in that case.
00:19:24.799 --> 00:19:25.839
But that's typically not the case.
00:19:25.920 --> 00:19:34.079
So agent observability traces, other platforms that people do their work in, using those to like seed the environments and the instructions.
00:19:34.160 --> 00:19:34.400
Yeah.
00:19:34.640 --> 00:19:41.519
And then synthesizing verifiers probably need a human in the loop to do like last mile, take those things over the finish line.
00:19:41.599 --> 00:19:42.799
So you actually have tasks.
00:19:43.039 --> 00:19:48.559
And then once you have tasks, in theory, like that becomes a verifier for your agent to start creating the automation itself.
00:19:48.720 --> 00:19:48.960
Right.
00:19:49.119 --> 00:19:50.720
And then that gets to put into production.
00:19:50.799 --> 00:19:53.759
You get production traces, you take those traces, you turn them into tasks.
00:19:53.839 --> 00:19:57.599
And hopefully it's like this is like the flywheel or the loop everyone talks about.
00:19:57.680 --> 00:20:00.400
But I do think that that is probably the North Star worth building towards.
00:20:00.640 --> 00:20:01.359
That's super exciting.
00:20:01.440 --> 00:20:05.839
To me, it's about making the way we program agents way more accessible in general.
00:20:06.000 --> 00:20:16.079
The more you can build verifiers with experts in the loop, of course, searing where the capabilities are going, the more you're able to shape these models in the ways that people actually want to use them or businesses want to use them.
00:20:16.160 --> 00:20:17.119
I think it's a great bet.
00:20:17.359 --> 00:20:18.480
Okay, last question.
00:20:18.720 --> 00:20:25.279
What do you feel is something that people don't tell you about building benchmarks that that is generally surprising?
00:20:25.680 --> 00:20:34.400
I think it's more fun to create a task than people realize, especially if it's a problem that you have solved historically that was hard for you.
00:20:34.640 --> 00:20:40.559
It's it's at least for me, it's a fun mental exercise to think, okay, what is the best way to frame this up in a way that's verifiable?
00:20:40.720 --> 00:20:42.640
And then I get to go see if the agents can solve it.
00:20:42.720 --> 00:20:44.240
So that's maybe an unexpected thing.
00:20:44.400 --> 00:20:50.319
And then the other unexpected thing is I think how painful it is to review tasks.
00:20:50.480 --> 00:21:00.079
It's extremely difficult to look at somebody else's task, try to load it into your brain, decide whether or not it's implemented well and what capabilities it's testing.
00:21:00.240 --> 00:21:09.039
So I think it's a lot more of a grind than people think to review the benchmark and put it out, but it's a lot more exciting than they expect to actually like create the tasks in the benchmark.
00:21:09.279 --> 00:21:12.720
Anything else you want to share more broadly that I didn't ask about?
00:21:13.039 --> 00:21:17.759
I guess the only other thing I'd mention is we're working on terminal bench three right now.
00:21:18.000 --> 00:21:23.279
And the thing we want to do most is this is what I was actually telling Vincent beforehand.
00:21:23.440 --> 00:21:28.480
Offline is with Terminal Bench 1, it was mostly we tried to create, and then we had to like pretty much beg people for tasks.
00:21:28.640 --> 00:21:31.680
And then we achieved some level of success based off of it.
00:21:31.920 --> 00:21:38.720
Terminal bench two is a little bit easier to get tasks for like other PhC students and stuff because it was becoming more popular for benchmark.
00:21:38.880 --> 00:21:43.519
But I think now we're in a position where Terminal Bench 3 can be.
00:21:44.160 --> 00:21:49.680
I think it's for a lot of people, it's their best opportunity to influence model development.
00:21:49.839 --> 00:22:04.799
And if we get enough buy-in from people who are great at making tasks that otherwise can go make a little mini benchmark themselves, but decide, okay, this should be part of terminal bench three, we can create like The best benchmark out there that should like dictate model development for the next year or something like that.
00:22:05.039 --> 00:22:14.720
So this means like trying to get the best people at the best companies to come and contribute a task of the most difficult thing that they've had to solve that they would be paid a lot of money to solve.
00:22:14.880 --> 00:22:20.799
So yeah, we're looking for contributors for Terminal Bench 3, and we would love to get as many people as possible involved in that process.
00:22:20.960 --> 00:22:21.359
Love it.
00:22:21.440 --> 00:22:26.559
So if you want to shape AI and where the field's going, Terminal Bench 3, contribute and you'll play a big role.
00:22:26.720 --> 00:22:27.119
Awesome.
00:22:27.200 --> 00:22:27.920
Thanks so much, Alex.
00:22:28.160 --> 00:22:29.440
Christian Bench Talks.
00:22:29.519 --> 00:22:29.759
That's right.
00:22:30.000 --> 00:22:31.440
We'll be excited to have you back soon.
00:22:31.519 --> 00:22:32.079
Thank you so much.
00:22:32.319 --> 00:22:33.039
Thanks.
