Benchtalks #2: John Yang (SWE-bench, ProgramBench) - The future of coding benchmarks

WEBVTT
00:00:00.239 --> 00:00:07.440
Honestly, when SWE-bench was released, I got some comments from I supportive, wonderful people, but at the release they were like, oh, this seems impossible.
00:00:07.440 --> 00:00:10.560
I don't know if models will ever be able to do GitHub, you know?
00:00:10.560 --> 00:00:23.120
And funnily enough, I think you get the same wave of comments think this kind of skepticism I think is incredibly helpful.
00:00:23.120 --> 00:00:25.518
And I think it's almost healthy.
00:00:25.518 --> 00:00:44.158
But to me, I think the reason I've gone in the direction that I And I think the reason I say why it's a big pivot now is I think we've very very much entering this realm of before you could sort of find things where it's like, well, you know, Vincent and and humans in general can do this task in models.
00:00:44.158 --> 00:00:54.320
And now, I mean, I can't implement FFmpeg from scratch in one And I think that's a gear shift that's meaningful.
00:00:54.320 --> 00:00:59.520
Yeah, it's challenging to get right, and I'm sure we'll get it have faith in that direct.
00:01:02.560 --> 00:01:06.959
Welcome to Benchtalks, Snorkel AI's podcast on benchmarks, quality, and real-world impact.
00:01:06.959 --> 00:01:22.000
Today, Snorkel co-founder Vincent Sunn Chen sits down with Yang, the creator of iconic software engineering the SWE-bench franchise, SWE-smith, CodeClash, and most ProgramBench.
00:01:22.560 --> 00:01:22.560
I'm here with John Yang.
00:01:22.560 --> 00:01:39.920
John is a PhD student at Stanford and creator of our favorite software engineering benchmarks, including the SWE-bench SuiteSmith, CodeClash, and most recently ProgramBench.
00:01:39.920 --> 00:01:44.319
we're super excited to have you.
00:01:44.319 --> 00:01:46.078
Welcome to Benchtalks, John.
00:01:47.680 --> 00:01:47.680
Thanks so much for having me.
00:01:49.200 --> 00:01:49.200
Last week you dropped ProgramBench.
00:01:49.200 --> 00:02:09.280
this was a really exciting new benchmark focused on end-to-end program level output, similar to Carlini's work on the C When you first launched it, every Frontier model had 0% pass I guess just this week, you know, GPT cracked that, you know, 0% what has the release been like?
00:02:09.280 --> 00:02:11.520
what what has the last week in general been like for you?
00:02:14.240 --> 00:02:14.240
Yeah, it's been a fun time.
00:02:14.240 --> 00:02:38.560
I think a lot of people were quite excited because in many ways, works that kind of have investigated this problem at a case And to some extent, I feel like it is sort of a natural to sort of the SWE-bench setting when you have sort of this just here's a GitHub issue, and then you create a PR for it.
00:02:38.560 --> 00:02:41.199
And that was meaningful for two years.
00:02:41.199 --> 00:02:46.080
But I think now, now that these models are so good at solving together a large application?
00:02:46.080 --> 00:02:53.199
And so, you know, Nicholas Carlini and Anthropic's blog on the inspiration.
00:02:53.199 --> 00:03:05.038
there's a blog post from Cursor that was talking about kind using to put together a browser.
00:03:05.038 --> 00:03:05.680
and then Epoch AI along with METR.
00:03:05.680 --> 00:03:17.038
they had RE-Bench where they did this case study on four that sort of captured a lot of the things that we were thinking So that was super exciting.
00:03:17.038 --> 00:03:34.877
to kind of TLDR, I feel like the role of ProgramBench is really to add sort of enough task instances such that we can study domain with meaningful statistical power.
00:03:34.877 --> 00:03:39.598
Because I think when you have these kind of one-offs, these don't necessarily transfer.
00:03:39.598 --> 00:03:40.877
People use different things.
00:03:40.877 --> 00:03:52.479
There's certainly sort of different incentives that people So I think as with all benchmark builders, the hope is like this kind of equal ground, equal footing where we can study task, you know, with purpose and sort of with a lot of clarity.
00:03:54.877 --> 00:03:58.479
Yeah, no, it's super exciting launch and overall kind of for where these software engineering style tasks are going.
00:03:58.479 --> 00:04:01.360
Talk to me a little bit more about the style of the eval.
00:04:01.360 --> 00:04:14.157
in this case, you're evaluating the artifact, not just you know, you're up-leveling the interface and Help me understand that a little bit more.
00:04:14.157 --> 00:04:15.840
What is the what is the goal with that specific task
00:04:18.720 --> 00:04:18.720
Yeah, I really love the description you just gave.
00:04:18.720 --> 00:04:21.759
I think I couldn't have said it better.
00:04:21.759 --> 00:04:31.040
one of the key things, I think, for me was there have generation benchmarks that existed before.
00:04:31.040 --> 00:04:50.319
so there's like really great work that Sasha Rush and his student which was released like close to around when SWE-bench was and so that paper was sort of a very good reference point and also there was we're gonna generate a Python library from scratch.
00:04:50.319 --> 00:04:59.600
They, you know, took I don't know, something like marshmallow or implementation, but they still kept kind of the function headers class headers and everything.
00:04:59.600 --> 00:05:10.160
And the way they evaluated was the models effectively asked to use the existing unit test suite.
00:05:10.160 --> 00:05:15.278
I think sort of the big thing I wanted to do was how do I just the model?
00:05:15.278 --> 00:05:34.079
Meaning that, you know, with CommitZero, you're still imposing a dictates what language the model should be implementing in, you what are the classes and the functions and the relationships and the typing.
00:05:34.079 --> 00:05:42.399
And so it's certainly not everything, but it's a very good of like system design and language selection and how do you about decomposing modules?
00:05:42.399 --> 00:05:58.879
And I think these are all things that we want to meaningfully So the point, sort of, sort of the small eureka moment was when I and he said, Oh, you know, like it's not so much the it's really the artifact, the deliverable that it creates.
00:05:58.879 --> 00:06:13.360
And so that was when we kind of made this decision of, all right, let's just completely put aside implementation as sort of a principle for the benchmark, and let's just do everything around the artifact.
00:06:13.360 --> 00:06:14.319
Or, you know, that's why it's called ProgramBench, the program.
00:06:14.319 --> 00:06:21.360
but specifically in this case, we do kind of executables and
00:06:23.278 --> 00:06:30.879
Yeah, I find that super interesting in your characterization know, the actual ways that coding agents are working.
00:06:30.879 --> 00:06:35.120
I think in the paper you had a few interesting findings, at to me, you know, there were flatter directories in general.
00:06:35.120 --> 00:06:40.959
I find it funny that it brings back up the kind of monolith, know, type of debate in software engineering.
00:06:40.959 --> 00:06:56.560
interestingly, you know, we saw GPT one-shot, you know, almost What were some of the surprising findings on your end of and more end-to-end types of problems?
00:06:57.920 --> 00:07:13.278
One I particularly love was that so in the default inference for ProgramBench when model solve it, the key constraints are you know, we don't allow them to use Ghidra or these kind of like engineering of the binary.
00:07:13.278 --> 00:07:22.639
So those are the two key constraints amongst the smaller but we do allow the model to implement its solution in any language.
00:07:22.639 --> 00:07:39.120
So instinctively, you might think that, oh, if the model, it's written in Rust and the model chooses Rust to kind of re-implement solution, it must have a bit of a leg up, you know.
00:07:39.120 --> 00:07:40.560
But what we found is that's not always the case.
00:07:40.560 --> 00:07:46.480
So first and foremost, models don't just kind of pattern match and FFmpeg.
00:07:46.480 --> 00:07:48.319
I'm definitely going to implement it in C.
00:07:48.319 --> 00:07:54.480
we have this kind of nice language confusion matrix in the shows this.
00:07:54.480 --> 00:08:20.800
I think the key takeaway that was quite fun for me was that models And I think this that that artifact of post-training is very, But what's more so is with GPT 5.5, which was the model that you mentioned that solved the very first task instance out of the 200 That that cmatrix animation was not written in Python, but the Python.
00:08:20.800 --> 00:08:31.838
And if you look at sort of the proportions, so I think the Claude models tend to kind of be more like sometimes I'll use Go or Rust, rarely C or C++ for anybody, which I think is interesting.
00:08:31.838 --> 00:08:46.000
I'm not sure how to interpret that takeaway, but yeah, I mean GPT Python, which initially you think maybe it's shooting itself in much not.
00:08:46.000 --> 00:08:46.320
So yeah, that's fascinating.
00:08:48.000 --> 00:08:57.360
I think one higher level reflection I've had, I mean, the headliner here is hey, you've built this really effective, you know, first of its kind benchmark, right, that that had a lot of headroom, right?
00:08:57.360 --> 00:08:58.080
The 0% you know, frontier model score at launch.
00:08:58.080 --> 00:09:08.879
but I will I really see this all as almost a research tool, We're able to ask a lot more questions about how to fuzz or you know, a lot of these repos as we're talking about.
00:09:08.879 --> 00:09:18.399
We're able to ask questions about, you know, cost, cost versus quality trade-offs, you know, in terms of implementation, different types of tasks, right?
00:09:18.399 --> 00:09:22.720
You know, how does performance different differ from features?
00:09:22.720 --> 00:09:29.278
how how do you think about this as a research tool or something that you know the community should be building on top of more broadly?
00:09:29.278 --> 00:09:29.278
Yeah, for sure.
00:09:29.278 --> 00:09:38.320
I mean, I think in the same way that when I read the cursor blog gave me like so much insight and information.
00:09:38.320 --> 00:09:56.960
we do 200, and I think that's the right way to go to sort of scale But I think even just the experience of like looking at what do just for this task instance, and you run multiple scaffold to mini sweet agent, right?
00:09:56.960 --> 00:10:01.360
which is sort of the justification being that this is the SWE-bench era.
00:10:01.360 --> 00:10:03.840
Right.
00:10:03.840 --> 00:10:04.080
And now we want to stress test it.
00:10:04.080 --> 00:10:12.240
And it's actually good, I think, that it starts a conversation scaffold can be better, right?
00:10:12.240 --> 00:10:30.798
But sorry, to answer your question very directly, I would just instance, like over-indexing on that, just looking at sort of this model does this, like what does the actual trajectory look What proportion of time is it spending actually probing the How often is it writing it?
00:10:30.798 --> 00:10:39.038
That's kind of where that insight of like, you know, cloud is probe a little bit, implement a little bit, right.
00:10:39.038 --> 00:10:43.120
And then kind of the next round of probing is to understand of what is the gap in what I have in progress in the reference.
00:10:43.120 --> 00:10:50.240
Versus GPT says, oh, you know, let me just do like a really job, right?
00:10:50.240 --> 00:10:51.120
Flesh out this very sort of thorough specification for myself.
00:10:51.120 --> 00:10:55.677
Right.
00:10:55.677 --> 00:10:56.240
And then let me one-shot it against the specification.
00:10:56.240 --> 00:11:10.080
And so I think it's just the tip of the iceberg in terms of the, you know, the way we can understand these models as what we want to garner and how we want to affect this kind of
00:11:12.879 --> 00:11:24.480
So what is your view on how these benchmarks should be used by, in dictating, you know, the changes in the field or or how folks are going to be approaching this discipline moving forward with in the loop, of course.
00:11:24.480 --> 00:11:27.600
Yeah, definitely.
00:11:28.320 --> 00:11:36.158
I think something that a lot of prior benchmarks, especially Bench, I think has done a really fantastic job of is, you community effort, right?
00:11:36.158 --> 00:11:50.000
Like I think this tagline that you can add your task instance to something that the frontier is evaluated on.
00:11:50.000 --> 00:12:05.600
big credit to Mike and Alex and Ludwig and obviously the folks So the way I kind of hope this message carries over, and certainly I'll try to operationalize it, is that yeah, whatever your favorite command line tool is, put that in a ProgramBench.
00:12:05.600 --> 00:12:12.480
You know, we'll we'll open up a leaderboard, but also I think add tasks.
00:12:12.480 --> 00:12:18.960
and I think just the whole pipeline of understanding how to for it, right?
00:12:18.960 --> 00:12:32.557
How to actually sort of, whether it's a prompting problem or training problem, understanding the failure points point it at recreating sort of or developing different programs.
00:12:32.557 --> 00:12:53.600
I just think there's a lot of general takeaways that will be sort but then also things that you can really that are really valuable in terms of the nuances of like, wow, for this application, you cmatrix, which is this kind of nice terminal matrix animation.
00:12:53.600 --> 00:12:55.440
Yeah.
00:12:55.440 --> 00:13:04.158
And you can see, wow, it actually deals really well with kind and I think that's a surprising finding that I wouldn't
00:13:04.639 --> 00:13:08.480
I'd love to talk a little bit about the lineage you know, the future of it.
00:13:08.480 --> 00:13:11.360
Yeah.
00:13:11.360 --> 00:13:11.600
you know, walk me through how you got here, right?
00:13:11.600 --> 00:13:20.480
Like a lot of the band is back together from ProgramBench as But you know, you started with SWE-bench, the SWE-bench of data sets.
00:13:20.480 --> 00:13:26.158
the community kind of really took that on and stride, and date.
00:13:26.158 --> 00:13:42.720
You know, you did a bunch of work on SWE-smith that we're synthesize tasks against environments, tried different well, the tournament style eval, and most recently kind of What does that journey in lineage look like for you and where do you see that going?
00:13:43.120 --> 00:13:46.557
Oh man, a lot of luck for sure, and a lot of thanks to the I've been able to collaborate with.
00:13:46.557 --> 00:14:18.158
But yeah, I guess if I were to construct a narrative in hindsight, yeah, the this journey kind of started back in when I was a and the first person I had really worked with, who I think I did ReAct and Tree of Thoughts and a lot of this really fundamental and so I guess to answer this question in terms of sort of, you lineage and where it's going and kind of the things that still fundamental principles.
00:14:18.158 --> 00:14:27.360
I think the fundamental things are that, you know, there's rigor, reproducibility.
00:14:27.360 --> 00:14:38.960
I think this is something that when I collaborated with Carlos on together the task instances and finding those kind of the 2,294 task instances.
00:14:38.960 --> 00:14:54.879
So finding those PRs and issues, it wasn't you know difficult We kind of got together in like June 2023 and we just kind of one day.
00:14:54.879 --> 00:14:56.639
Right.
00:14:56.639 --> 00:14:56.879
And we spent like the next month and we had it.
00:14:56.879 --> 00:15:06.320
Really, what ended up taking us the next three or four months was sounds like amazing to people now, right?
00:15:06.320 --> 00:15:13.600
But when we released the SWE-bench originally, we didn't use that ended up being kind of a nightmare for reproducibility.
00:15:13.600 --> 00:15:23.677
this is totally my fault, like because Carlos was like, maybe we I was like, oh, but it's so heavy, there's so much, like you to download gigabytes of images.
00:15:23.677 --> 00:15:25.038
Let's just do conda, it's gonna be fine.
00:15:25.038 --> 00:15:27.360
Yeah, he was 100% right at the end of the day.
00:15:27.360 --> 00:15:30.720
so that was kind of a driving theme.
00:15:30.720 --> 00:15:39.759
And I think like Kilian and I invested so much time in making sure people could download it and run it.
00:15:39.759 --> 00:15:40.720
So that's kind of sort of what's remained the same.
00:15:40.720 --> 00:15:48.158
I think what has remained different is to be willing to our notions of what language models can do.
00:15:48.158 --> 00:15:56.320
And I think ProgramBench in particular to me represents a fairly big pivot.
00:15:56.320 --> 00:15:56.798
Right.
00:15:56.798 --> 00:16:13.840
So the very, very kind of first project I did was called Inner And this was just kind of taking a lot of like NL to bash, and MBPP and HumanEval, and these the a lot of these code benchmarks be sort of input-output, no interaction at all, right, and it into an interactive format.
00:16:13.840 --> 00:16:15.360
Right.
00:16:15.360 --> 00:16:15.759
And so the question there was can the model benefit from this?
00:16:15.759 --> 00:16:27.918
With SWE-bench, it was like, look, this stuff is not realistic, passing LeetCode questions, but this is not what we do day to Can models do this?
00:16:27.918 --> 00:16:37.600
And right honestly, when SWE-bench was released, I got some from I won't name who they're all very supportive, wonderful But at the release, they were like, oh, this seems impossible.
00:16:37.600 --> 00:16:40.320
I don't know if models will ever be able to do GitHub, you know.
00:16:40.320 --> 00:16:52.639
And funnily enough, I think you get the same wave of comments think this kind of skepticism I think is incredibly helpful.
00:16:52.639 --> 00:16:55.200
And I think it's almost healthy.
00:16:55.200 --> 00:17:13.200
But to me, I think the reason I've gone in the direction that I And I think the reason I say why it's a big pivot now is right, I realm of before you could sort of find things where it's like, and John and humans in general can do this task, can models.
00:17:13.200 --> 00:17:24.000
And now, I mean, I can't implement FFmpeg from scratch in one And I think that's a gear shift that's meaningful.
00:17:24.000 --> 00:17:29.200
Yeah, it's challenging to get right, and I'm sure we'll get it have faith in that direction.
00:17:29.200 --> 00:17:32.400
Yeah.
00:17:32.960 --> 00:17:32.960
I mean I completely agree.
00:17:32.960 --> 00:17:43.117
I think one of the things I find really exciting about the you and other, you know, benchmark builders have taken is for where the where the field is going, right?
00:17:43.117 --> 00:17:50.160
You're putting forward these challenges, you're you're a path where, hey, this is where we think capability should or or can go.
00:17:50.160 --> 00:17:56.640
And you know, the skepticism is often a good sign that orthogonal to what people believe is possible.
00:17:56.640 --> 00:18:04.400
But I think that ethos is really, really inspiring and an way, you know, for all people building benchmarks to you think about their work.
00:18:05.200 --> 00:18:05.200
Definitely, definitely.
00:18:05.200 --> 00:18:06.000
Yeah.
00:18:06.798 --> 00:18:10.557
I'd love to talk also a bit about the methodological differences of the benchmarks that you've worked on.
00:18:10.557 --> 00:18:23.597
You've explored a number of different ways to grade model from unit tests to you know tournament style eval, now to kind of this awesome fuzzing and you know output-driven validation approach.
00:18:23.597 --> 00:18:24.160
what have you learned about how to grade these models?
00:18:24.160 --> 00:18:27.278
What are the trade-offs between them?
00:18:28.798 --> 00:18:28.798
how do you think about this now?
00:18:28.798 --> 00:18:34.077
The evolution of verification itself, I think, is a very problem.
00:18:34.077 --> 00:18:38.400
And in a lot of ways, kind of drives the way I think about it.
00:18:38.400 --> 00:18:38.557
And I think a lot of people think about it.
00:18:38.557 --> 00:18:53.680
So I guess with SWE-bench, I think the verification, it didn't do MBPP, HumanEval apps, like all these kind of LeetCode style things, they also had unit tests as well.
00:18:53.680 --> 00:19:01.200
The change there was like, okay, with HumanEval, even in the actually have people write the tests.
00:19:01.200 --> 00:19:06.400
And right, Carlos and I were like, we don't, we're not open AI, How do we look for tests in the open, in the wild?
00:19:06.400 --> 00:19:09.357
Right.
00:19:09.357 --> 00:19:09.759
You know, so I think that was kind of the key thing there.
00:19:09.759 --> 00:19:19.680
Otherwise, from like a formulation standpoint, I think it why I'm sure it got adopted.
00:19:19.680 --> 00:19:24.640
With CodeClash, I think I was very kind of inspired by the Ella people have done.
00:19:24.640 --> 00:19:27.759
Right.
00:19:27.759 --> 00:19:44.077
and I thought, okay, what does it mean to kind of bring this into a There were some prior works that had done things like, you know, Copilot Arena or even in the products with VS Code and Cursor some voting mechanism of proposed version A, proposed version Right.
00:19:44.077 --> 00:19:57.278
there was something where I felt like that made sense, but I scale because like how many times are you gonna be able to ask A or B, where it's like two different code snippets?
00:19:57.278 --> 00:19:59.440
Right.
00:19:59.440 --> 00:20:03.117
Before they kind of throw up their hands and they say, I don't As long as it works.
00:20:03.117 --> 00:20:04.240
Right.
00:20:04.240 --> 00:20:04.557
You know, right.
00:20:04.557 --> 00:20:13.200
With CodeClash, it was kind of testing this idea of like, all code artifacts and make them compete against each other.
00:20:13.200 --> 00:20:14.557
Right.
00:20:14.557 --> 00:20:19.117
and so that would that then it landed itself into this idea very competition.
00:20:19.117 --> 00:20:29.038
I think something I still like about CodeClash, even though it ProgramBench, is that technically it deals with the little bit better.
00:20:29.038 --> 00:20:32.480
Right.
00:20:32.480 --> 00:20:32.798
Right?
00:20:32.798 --> 00:20:33.117
Like it's very open-ended.
00:20:33.117 --> 00:20:34.880
It's it's truly faithfully open-ended in that way.
00:20:34.880 --> 00:20:45.519
You're not gonna run into like, if a model is able to create solution that's better than another, that's all that's needed Right.
00:20:45.519 --> 00:21:17.920
so I think like I'm hoping that maybe it's like a little bit It's also possible that because the arenas are all games, sort of, I want more economically valuable things, which But I think that idea of making artifacts compete against inspiration for ProgramBench, which is okay, it's still sort of factor-wise written as a pytest, but they're all invocations it's all calls of the program.
00:21:17.920 --> 00:21:20.880
It is no longer touching anything about the implementation.
00:21:20.880 --> 00:21:29.597
So now you completely disentangle the evaluation from sort Right.
00:21:29.597 --> 00:21:36.240
You know, the overarching theme is sort of like fundamentally I think with SWE-bench, it was can you implement correctly?
00:21:36.240 --> 00:21:46.240
With CodeClash, it's hey, can you kind of evolve your code base Right.
00:21:46.240 --> 00:21:46.557
And then with ProgramBench, it's can you create this artifact?
00:21:46.557 --> 00:21:49.357
Yeah.
00:21:49.357 --> 00:21:59.519
And I think these like from that sort of what's the end goal of view, I think then that will kind of naturally sort of with Inform sort of what the verification should look like.
00:21:59.519 --> 00:22:02.640
Yeah.
00:22:03.038 --> 00:22:03.038
That that resonates.
00:22:03.038 --> 00:22:21.597
The hey, what is the outcome that I'm trying to actually And then, you know, let's let's find a clever way to actually, you know, inject that reward or learning signal as a Where do you see as the role of human knowledge or human in driving these evaluations or steering these types more broadly?
00:22:21.597 --> 00:22:27.759
I know you had a position paper, you know, around how humans these coding benchmarks and research in general.
00:22:27.759 --> 00:22:32.400
Where do humans need to be in the loop as you're thinking about these types of mechanisms?
00:22:33.200 --> 00:22:37.117
I've had a couple conversations with my advisor, Diyi very into like human AI collaboration.
00:22:37.117 --> 00:22:54.720
And I kind of came into the PhD program being sort of more But in the two years since I've been very convinced that like And so to really sort of ground it, the inspiration, Program SWE-bench for sure.
00:22:54.720 --> 00:23:02.960
and ultimately it's exciting that the solution space is so large.
00:23:02.960 --> 00:23:10.317
But in some sense, like even as human software developers, like have certain characteristics to them.
00:23:10.317 --> 00:23:11.278
And it's kind of conditioned on the purpose of the software.
00:23:11.278 --> 00:23:17.278
But generally, you know, we want them to be styled well, we want them to be readable, we want them to, you know, be portable.
00:23:17.278 --> 00:23:42.720
I think there's some general heuristics that are true, where I'm sure some of these labs are probably already post-training like this, but something about sort of like empowering the individual to say things and then very quickly steer the model effectively their preferences.
00:23:42.720 --> 00:23:43.117
Right.
00:23:43.117 --> 00:23:48.317
I think generally it's just important to me this idea that want it built.
00:23:48.317 --> 00:23:50.960
I don't understand hardware.
00:23:50.960 --> 00:23:52.480
I don't understand what goes into like HDL.
00:23:52.480 --> 00:23:54.798
Those people are experts.
00:23:54.798 --> 00:23:59.920
But as you and I, as the people who are putting these tools it's not sort of my.
00:23:59.920 --> 00:24:10.960
I think a better way to kind of solve the problem is not to kind of figure out what they do and then decide kind of like, all right, these are the things we should inject, but more to just make the model itself fundamentally steerable.
00:24:10.960 --> 00:24:24.960
So in the position paper, like me and Zora, who's a wonderful Neubig and the OpenHands folks who are great, she talks about this idea of steerability in the paper.
00:24:24.960 --> 00:24:42.077
And I think that's something that feels very compelling to me of the day where we're at like 80% on ProgramBench, that's very But then what does that mean for the people that ultimately use these things?
00:24:42.077 --> 00:24:44.557
Right.
00:24:44.557 --> 00:24:54.880
If it can get 80% on ProgramBench, but it's just and it's not really using a file system and the code is not Yeah.
00:24:54.880 --> 00:25:02.000
that's like it's a meaningful end state, but I think we can ask Yeah.
00:25:02.557 --> 00:25:02.557
Yeah, I agree with that.
00:25:02.557 --> 00:25:07.597
I think that that end state is more ambitious and harder to kind general.
00:25:07.597 --> 00:25:15.920
But yeah, we're really glad that you put out that position paper charting of again, kind of where we think things might go.
00:25:15.920 --> 00:25:17.680
I'd love to talk a bit about quality as well.
00:25:17.680 --> 00:25:39.200
I think one of the, you know, pieces that we you know pay a building data sets or environments or benchmarks is hey, overall aggregate quality, you know, is at a really high individual task level quality is at, you know, high you know, having models or agents in the loop is really It can be, you know, a double-edged sword in some cases.
00:25:39.200 --> 00:25:43.357
Yeah.
00:25:43.357 --> 00:25:52.160
in the case of you know, ProgramBench and you know, where of steer, you know, generate a bunch of the kind of instances.
00:25:52.160 --> 00:26:05.200
You've had a lot of learnings from you know, SWE-smith where you know generated tasks from well-curated How do you think about these settings where you know agents loop?
00:26:05.200 --> 00:26:10.720
but you know, you also want to manage really, really high
00:26:11.357 --> 00:26:25.117
So far, to be honest with SWE-smith and ProgramBench, it's a funny thing with benchmarking where it's like you effectively models can't quite do yet that you want and you want to help them get there.
00:26:25.117 --> 00:26:28.640
But along the way, if there are certain portions that the model engaged, you know.
00:26:28.640 --> 00:26:31.200
Of course.
00:26:31.200 --> 00:26:31.519
So it's it's kind of funny.
00:26:31.519 --> 00:26:41.038
But with ProgramBench, I think the big thing was that test suites are not available, like behavioral style test suites that invocations of these programs are not available.
00:26:41.038 --> 00:26:56.077
empirically, we could ask a person to kind of help with this, and to some extent, like I'm not sure that a person without true can really write a good test suite.
00:26:56.077 --> 00:27:07.680
To concretely answer this question, I think models these days that there's probably three vectors I would imagine you could using the model.
00:27:07.680 --> 00:27:08.317
Right.
00:27:08.317 --> 00:27:08.640
One is to assist you like with generating the verification.
00:27:08.640 --> 00:27:11.278
Right, right, right.
00:27:11.278 --> 00:27:23.519
One of them, so the SWE-smith-oriented take is to task instances, which is like you have the code base, it's and then you do things like you know, you ask the model to you know, kind of a funky implementation that breaks tests.
00:27:23.519 --> 00:27:26.077
Right.
00:27:26.077 --> 00:27:26.400
And that's another thing.
00:27:26.400 --> 00:27:30.798
The last factor I would say is actually constructing the itself.
00:27:30.798 --> 00:27:56.000
So this is something where for both SWE-smith and ProgramBench, it Where for ProgramBench, it was, okay, if I want to get the reference executable, clone the code base and then ask the model to actually write the compilation script with all of the With SWE-smith, it was here's the Python library, make sure you sure you can run the unit tests as well.
00:27:56.000 --> 00:28:08.480
So I think sort of it's hard for me to come up with like a good mainly because like I think people are just experimenting with and it's just empirically in effective.
00:28:08.480 --> 00:28:10.557
Yeah, yeah, yeah, yeah.
00:28:10.557 --> 00:28:11.597
but yeah.
00:28:12.400 --> 00:28:15.117
No, I don't think there's a one size fits all solution either, right?
00:28:15.117 --> 00:28:29.200
I mean, in practice, a lot of what I and our teams find is you expertise and steering and and you know, high-level guidance in right places efficiently is it's tricky, it's case by case, you know, it's pipeline or workflow dependent.
00:28:29.200 --> 00:28:29.759
Yeah.
00:28:29.759 --> 00:28:36.880
but you know, having an intelligent way to say, hey, this is you know, to steer this part of the program.
00:28:36.880 --> 00:28:40.720
This is where actually, yeah, let the agents rip and start take it over.
00:28:40.720 --> 00:28:41.038
I think makes a lot of sense.
00:28:41.038 --> 00:28:45.357
so you know, you're you're certainly at the frontier of that, which is why I thought it was a yeah.
00:28:47.920 --> 00:28:51.117
I mean, I think like in some sense, we can afford to kind of be small scale experimentation.
00:28:51.117 --> 00:28:54.720
Right.
00:28:54.720 --> 00:28:58.720
Because unlike before, once you get that right, it's much easier to scale up with agentic systems.
00:29:00.400 --> 00:29:08.240
In the ProgramBench paper, you stated that models that had hacking up to, I think, 36% of the time.
00:29:08.240 --> 00:29:13.117
You've also found, you know, back back in your SWE-bench days, right?
00:29:13.117 --> 00:29:17.519
let's call it leaderboard hacking when it came to you know how do you think about leaderboards, open benchmarks?
00:29:17.519 --> 00:29:28.077
and as a benchmark builder, you know, how to address this that we're measuring model performance in an honest and
00:29:30.798 --> 00:29:30.798
I think there's like two parts to this really great question.
00:29:30.798 --> 00:29:36.798
The first half, which is like models are cheating, and the second, which is like, how do you set up a leaderboard that's open?
00:29:36.798 --> 00:29:39.357
And they're both really meaningful questions.
00:29:39.357 --> 00:29:49.038
When ProgramBench was launched, I think a big topic online internet is certainly useful.
00:29:49.038 --> 00:29:49.038
so why disallow it?
00:29:49.038 --> 00:30:06.960
and I would say like the central thesis that we were really anytime there's X percent improvement on ProgramBench, we want be sort of like undoubted, that there's no question, there's no model actually got it.
00:30:06.960 --> 00:30:12.000
It would be really cool to see a model sort of go on GitHub and source code itself.
00:30:12.000 --> 00:30:15.440
Right.
00:30:15.440 --> 00:30:15.920
and sort of do these things.
00:30:15.920 --> 00:30:26.317
Maybe it checks Stack Overflow, maybe it checks like a I agree that the internet is useful, but not at the cost of the rigor of the benchmark.
00:30:26.317 --> 00:30:27.759
And so the finding that you pointed out, it was twofold.
00:30:27.759 --> 00:30:47.278
We early on in the experimentation process, we allowed And then what we would do, and this is big credit to Kilian, is run at the trajectories and then they decide like did the model cheat Right.
00:30:47.278 --> 00:30:51.038
And you have these kind of cheating criteria specified in language.
00:30:51.038 --> 00:31:05.200
The problem wasn't just that there was, I mean, like a third is very tricky for us to release a benchmark where we're saying zero percent, but also like they cheat on like the messaging is very Right.
00:31:05.200 --> 00:31:05.519
And Kilian also found that the judges would disagree.
00:31:05.519 --> 00:31:21.920
And we have like a couple examples in the paper where like saying, like, oh, it's not allowed to look it up on SourceForge And the four is saying, well, it was only GitHub that was disallowed, so SourceForge should be fine.
00:31:21.920 --> 00:31:24.720
Right.
00:31:24.720 --> 00:31:25.038
And it becomes this kind of crazy cat and mouse game.
00:31:25.038 --> 00:31:35.680
So I think this is a constant tug of war for benchmark builders, but the typical advice I would go for is that rigor is king, is king, reproducibility is king.
00:31:35.680 --> 00:31:38.240
Everything else is kind of cherry on top.
00:31:38.240 --> 00:31:51.597
But if any of those things start to really affect, you know, the rigor is kind of really core to being able to then, you have people come together and hill climb and not point at each didn't do it right.
00:31:51.597 --> 00:31:53.759
You know, that's important.
00:31:55.038 --> 00:31:55.038
For the second half of setting up the leaderboard, yeah, I
00:31:58.400 --> 00:32:03.440
curating submissions for SWE-bench was certainly like a a wonderfully interesting experience.
00:32:03.440 --> 00:32:13.038
like as of today, and I we've kind of you know tailed off on I think we ended up accepting like over 300 on the leaderboard.
00:32:13.038 --> 00:32:27.519
I think the experience there is to set up leaderboard submission pipelines that enable members of the community to kind of be by and also sort of like double check each other's work.
00:32:27.519 --> 00:32:30.400
Right.
00:32:30.400 --> 00:33:00.000
So there was a big moment in SWE-bench where because of sort of has been repeated in later benchmarks, that we said you must And I'm really glad to say I think this is something that Bench has adopted now, and we'll certainly see that adopted for But the moment we ask people for all those trajectories, one, I submit, I think naturally, you know, you kind of like remove those solutions came to be.
00:33:00.000 --> 00:33:13.357
And then the second thing is when the trajectories get uploaded, that people can look at it and they they can kind of it does inspire new approaches, but it also does help with sort of questions of is this cheating?
00:33:13.357 --> 00:33:29.759
You know, like like the credit to the fair coding team, they were behavior of SWE-bench like of some of the models fast forwarding, like going to future commits to solve the problem.
00:33:29.759 --> 00:33:35.920
And thanks to them, because after that we realized, you know, a reporting these things, right?
00:33:35.920 --> 00:33:54.480
Not necessarily out of malice, maybe because they just hadn't but yeah, I think just as open as possible because it just helps reproducibility, with kind of trust that you can build with, you you as a leaderboard host and the people participating in that.
00:33:54.480 --> 00:33:56.317
Yeah.
00:33:56.317 --> 00:33:56.640
yeah, never bad to be too open.
00:33:59.038 --> 00:33:59.038
That makes sense.
00:33:59.038 --> 00:33:59.920
How do you go about that error mode, failure mode analysis?
00:33:59.920 --> 00:34:02.880
I can only imagine, right?
00:34:02.880 --> 00:34:08.239
As these trajectories get longer and longer horizon, as the as you pointed out, get more and more creative.
00:34:08.239 --> 00:34:14.239
how does one try to understand and even kind of dig into
00:34:16.320 --> 00:34:26.400
I think I there's like two sides of the coin to me for me on One side is like, we should definitely invest in sort of better Using language models themselves.
00:34:26.400 --> 00:34:27.920
I think there's been great research around this.
00:34:27.920 --> 00:34:38.800
Like you know, for instance, the Transluce folks have the Docent inspect SWE-bench trajectories using Docent.
00:34:38.800 --> 00:34:39.760
They're all loaded up.
00:34:39.760 --> 00:34:49.360
You can ask questions about it, you know, and it'll it'll do of give you an answer aggregated across all of the So tools like that, absolutely worth investing in.
00:34:49.360 --> 00:34:56.800
But then on the other hand, nothing beats just like sitting down and like manually scrolling through one of them.
00:34:56.800 --> 00:34:59.039
Yeah, yeah.
00:34:59.039 --> 00:34:59.440
So I think it's kind of like a balance.
00:34:59.440 --> 00:35:15.119
If anything, it kind of echoes the thing we said earlier about like human involvement in these things of just, yeah, take a little time, figure out how it looks for one or two instances manually, and then you know, scale to you know above and beyond.
00:35:17.440 --> 00:35:17.440
Yeah, I'm a I'm a big believer in both as well.
00:35:17.440 --> 00:35:23.119
We used to have as an onboarding exercise at snorkel, you gotta dig into the data, dig into the trajectories.
00:35:23.119 --> 00:35:32.320
Because to your point, there's nothing, nothing beats personal firsthand intuition for what the shape of the like, you know, what artifacts might look like, right?
00:35:32.320 --> 00:35:38.320
And you know, then translating that to more You know, I agree with that method.
00:35:38.320 --> 00:35:39.840
Yeah.
00:35:39.840 --> 00:35:40.159
It makes a lot of sense.
00:35:40.960 --> 00:35:45.920
Yeah, there's just the threat of drift if it's always kind of one Exactly.
00:35:46.320 --> 00:35:46.320
I'd love to move to a lightning round, if you don't mind.
00:35:46.320 --> 00:35:48.800
So you know, hot takes are welcome.
00:35:48.800 --> 00:35:57.360
you know, no need to go super, super deep, but let's just these high-level ideas.
00:35:57.360 --> 00:36:11.599
You tweeted that we're we're moving from a regime where you know, the X things that humans can do, but we're shifting to we're asking the question, can language model actually do that were previously impossible?
00:36:11.599 --> 00:36:12.800
what are the types of tasks that fall into that second category?
00:36:12.800 --> 00:36:16.400
Yeah.
00:36:17.440 --> 00:36:25.360
I've talked with Ofir Press, who's like a longtime collaborator some really excellent thoughts.
00:36:25.360 --> 00:36:35.599
He has this one diagram which is called like the five steps or where like one, two, three are kind of one, two, three are all kind of in the realm of like, can humans do this, you know?
00:36:35.599 --> 00:36:45.199
but then four and five are kind of like, oh wow, humans can't So I think ProgramBench, and this is kind of very much stealing So thanks, Ofir.
00:36:45.199 --> 00:36:49.599
But four is sort of in the realm of like there is kind of a solution that exists.
00:36:49.599 --> 00:36:54.400
So for example, for ProgramBench, the FFmpeg source code Right.
00:36:54.400 --> 00:36:54.960
you can generate tests on top of it.
00:36:54.960 --> 00:36:57.440
There's continuing to be progress on it.
00:36:57.440 --> 00:37:03.840
Basically, we're now compressing the timeline with which we can do that.
00:37:03.840 --> 00:37:11.679
So I think there's almost sort of models can effectively do the span than people can do.
00:37:11.679 --> 00:37:17.840
And so I think that's sort of superhuman in the sense of like, this off, but it just took us a way longer time.
00:37:17.840 --> 00:37:18.960
Right, right.
00:37:18.960 --> 00:37:51.518
so I think that's one way to think about it where it's like, what general, you know, representations of, you know, intelligence or, you know, you know, things that took a long, long that you can sort of now have the model try to recreate, like such a good proxy where it's like if a model could meaningfully that maybe it's gonna create equally revolutionary game-changing software that's novel, that's not FFmpeg that we haven't seen.
00:37:51.518 --> 00:37:53.280
Right.
00:37:53.280 --> 00:37:53.599
You know, but it's hard to grade it on those unseen things.
00:37:53.599 --> 00:37:56.480
So I think the seen things are good.
00:37:56.480 --> 00:38:30.079
The yeah, I think the fifth realm, and I think this is maybe and there's maybe people with like better formal ways to approach this, is like, yeah, just things that we know that are it's kind of those crazy Millennium Prize things, like and physics, like really challenging the boundaries of kind And I'm sure this is kind of where a lot of the excitement around you know, recursive self-improvement and AI scientists And I believe with that, I agree with that.
00:38:30.079 --> 00:38:46.400
I think my personal approach is more like having models be able that exist is a good stepping stone to then having meaningful something then new that's you know, just as, if not more
00:38:50.880 --> 00:38:50.880
Yeah, I subscribe to that mindset as well.
00:38:50.880 --> 00:38:53.360
I think there's a real value in empiricism, right?
00:38:53.360 --> 00:38:56.239
That's what benchmarks give us, right?
00:38:56.239 --> 00:39:10.559
I'm yeah, I am as much a fan as the kind of vibe eval or kind of you know, future looking more per projection-based eval, but scientifically study some of these models in the current you know, I think there's nothing that beats that.
00:39:10.559 --> 00:39:12.239
Exactly.
00:39:12.239 --> 00:39:12.719
Exactly.
00:39:13.199 --> 00:39:13.199
Couldn't agree more.
00:39:13.199 --> 00:39:14.320
Yeah.
00:39:14.320 --> 00:39:14.559
Awesome.
00:39:15.039 --> 00:39:21.119
so GPT-5 just cracked, you know, one of the tasks in when does the first model hit 80%?
00:39:23.679 --> 00:39:23.679
80%.
00:39:23.679 --> 00:39:24.880
Yeah, I guess for our sakes, hopefully not too soon.
00:39:24.880 --> 00:39:30.639
Just joking.
00:39:30.639 --> 00:39:32.159
maybe like a year, maybe like a year and a half from now.
00:39:32.159 --> 00:39:46.480
the way I'm thinking about it is with with ProgramBench, there is of nicely happened to have a gradient of difficulty.
00:39:46.480 --> 00:39:47.760
Right.
00:39:47.760 --> 00:39:53.119
So, you know, cmatrix, we have this kind of like difficulty scoring thing that we put in the appendix.
00:39:53.119 --> 00:39:58.719
If you look at cmatrix, it's it's literally one of the easiest And our difficulty rating is not empirically based.
00:39:58.719 --> 00:40:05.360
It's purely just based on looking at lines of code and the of dependencies.
00:40:05.360 --> 00:40:09.920
It's purely grading with respect to the characteristics of So it's totally independent of performance.
00:40:09.920 --> 00:40:18.880
And why I think that's meaningful is yeah, you know, the it's solved, it's fantastic, but you know, it's one of the easier ones, you know.
00:40:18.880 --> 00:40:41.760
So I think there's this nice grading of difficulty really at the of the toughest task instances, SQLite, the PHP there's this one really amazing dev called Fabrice Bellard, really fantastic software, like FFmpeg, whatever.
00:40:41.760 --> 00:40:55.199
I think there's just, I mean, there's a lot of just like true and I'm pretty, I have faith that the models will sort of become, one day.
00:40:55.199 --> 00:41:08.719
I think 80% to me on ProgramBench, which is like 160 out that sort of like start are quite meaningful at that point.
00:41:08.719 --> 00:41:11.119
Right.
00:41:11.119 --> 00:41:15.280
in terms of, okay, maybe it gets things like ripgrep or search in their utility.
00:41:15.280 --> 00:41:18.239
the remaining 20%, I think it I'm not sure.
00:41:18.239 --> 00:41:23.920
Maybe it doesn't take that long.
00:41:23.920 --> 00:41:31.360
My my sort of hunch, at least for a human, is that you know it's a little bit of a long tail in that, you know, something like they're tricky.
00:41:31.360 --> 00:41:37.920
Yeah, but I mean compared to FFmpeg, this is kind of a different leak.
00:41:37.920 --> 00:41:43.360
So I think, you know, one year is kind of my my more bullish take, but I'm happy to be proven wrong.
00:41:43.360 --> 00:41:45.280
How how would you characterize?
00:41:46.960 --> 00:41:54.079
You said there's a meaningful difference between the newer and I like do do you have words to capture what what that shift actually looks like?
00:41:54.079 --> 00:41:55.518
Oh, for sure.
00:41:56.400 --> 00:41:59.599
I mean, I think some of these executables, like if you just see some of them have like a lot of subcommands.
00:41:59.599 --> 00:42:35.119
And so, in some sense, like even putting aside the lines of code and dependencies and all that, they just encompass different of sort of functionality, you know, like that FFmpeg handles so and it does all sorts of different kinds of, you know, you you know mp4, but you can apply this kind of like encoding to a model, you know, they're gonna really have to do some heavy downloading different assets.
00:42:35.119 --> 00:42:40.480
These assets vary by how long that video is, kind of what the person?
00:42:40.480 --> 00:42:48.400
Or, you know, I mean, I'm just coming up with these dimensions on things I've missed that FFmpeg accounts for.
00:42:48.400 --> 00:43:02.960
and so I think to some extent, ProgramBench beyond just correctness and models as software developers, I think it's extent of like, did you probe enough?
00:43:02.960 --> 00:43:21.360
You know, like because if you didn't probe enough, it doesn't And I think something about that almost extends to models then better sort of, you know, innovators in that so much of what there's no specification, there's no kind of well-written doc it.
00:43:21.360 --> 00:43:28.480
It's just someone really kind of poking endlessly at this thing that wasn't realized before.
00:43:28.480 --> 00:43:33.199
and they formalize it.
00:43:33.199 --> 00:43:38.239
You know, this is the whole like it's not a matter of just you enough.
00:43:38.239 --> 00:43:42.960
Like he just literally had to try a bunch of different metals know.
00:43:42.960 --> 00:43:48.960
So I think this kind of like persistence I think is exciting, you
00:43:49.599 --> 00:43:58.400
I love that notion that a bunch of software is actually the contacting reality and and really understanding.
00:43:58.400 --> 00:44:05.039
Understanding their own preferences, being curious and to really capture all the different ways that they might be to use it.
00:44:05.039 --> 00:44:14.400
And for something like ProgramBench to be able to extend that form of, you know, as you put it, curiosity or I think is a really exciting way to think about, you these models are going.
00:44:14.400 --> 00:44:15.119
Yeah.
00:44:15.119 --> 00:44:15.360
What benchmark should more people be paying attention to?
00:44:18.639 --> 00:44:28.639
I think these days my personal preferences and focuses are on long horizon things within the realm of kind of 20 to 30 to 40 SWE-bench.
00:44:28.639 --> 00:44:45.679
I think we have the formulas in place to kind of solve that so in terms of long horizon, I think it's meaningful because it know, I guess continual learning is a hot word these days.
00:44:45.679 --> 00:44:47.280
So I'll lean on that.
00:44:47.280 --> 00:44:53.119
but very much in the sense of like, how do models construct memory And that memory, it doesn't just have to be natural language.
00:44:53.119 --> 00:44:56.320
It could even be sort of like this model and this model.
00:44:56.320 --> 00:44:59.920
They both solve this task instance.
00:44:59.920 --> 00:45:04.800
But which one provides a solution that's more extensible to picks it up next?
00:45:04.800 --> 00:45:05.760
You know?
00:45:05.760 --> 00:45:15.280
so I think concretely, kudos to you guys, like the continual that your your team did, that feels like really great.
00:45:15.280 --> 00:45:34.639
I almost kind of like it in that the way I see it is that you like a hundred turns or you know, just as a random number, but learning bench, which is also effectively a hundred turns, but you middle depending on kind of like where things pick up.
00:45:34.639 --> 00:45:54.000
Like one of the tasks I really like was it's kind of resembles like where you have issue A, and you can kind of find these naturally occur online of like B was blocked by A, but then C was blocked and then obviously you can keep going there.
00:45:54.000 --> 00:46:06.559
And so once you unblock A, when you solve B, I mean, it's one that function in a way where it's kind of at the right layer in such that B can correctly invoke it, right?
00:46:06.559 --> 00:46:16.639
Versus, oh, you know, the solution is too high up in the call So then when you implement B, you end up redundantly repeating below.
00:46:16.639 --> 00:46:19.119
that seems really fascinating to me.
00:46:19.119 --> 00:46:21.920
Yeah.
00:46:21.920 --> 00:46:22.239
Yeah.
00:46:22.480 --> 00:46:22.480
Well, appreciate the kind words.
00:46:22.480 --> 00:46:23.679
Yes.
00:46:23.679 --> 00:46:33.440
A lot of the goal there was, hey, how do we measure an ability to learn from experience, learn from, you know, interacting with an environment, multiple instances environment.
00:46:33.440 --> 00:46:43.119
So it's something that humans can do quite efficiently and And I agree that that's that's you know one of the ways that know, autonomy long term.
00:46:43.119 --> 00:46:43.440
definitely.
00:46:43.440 --> 00:46:44.159
What is a benchmark that you wish existed?
00:46:44.159 --> 00:46:46.880
Oh man.
00:46:46.880 --> 00:46:48.880
I think there's there's a lot of them.
00:46:52.159 --> 00:46:57.199
One I think I'm fairly excited about is using coding generally to aren't just coding.
00:46:57.199 --> 00:47:13.199
So what I mean by that to expand is for instance, SWE-bench, I grade models on their ability to write code for the sake of better You know, we fix this bug.
00:47:13.199 --> 00:47:23.760
But I think as we've seen with like Claude Code and just like the proliferation of these kind of SWE-agents that people are code, right?
00:47:23.760 --> 00:47:38.480
Like in some sense, like I mean, Claude Cowork is fantastic, an engine where underlying the engine is very much just a coding agent, you know, with a couple extra tool calls and whatever.
00:47:38.480 --> 00:47:44.079
So I think I would just be really excited about leaning on the outside of.
00:47:44.079 --> 00:48:06.719
I mean, this is this is maybe speaking in too general, you know, too many generalities, but I truly believe like, hey, if you had sort of law problems, could you recast that into something you know, it's manipulating code to some extent, it's like through a bunch of documents, but you know, it builds its own So I guess this is like a very, very high-level answer.
00:48:06.719 --> 00:48:26.079
if I had to kind of sit down today and brainstorm and start the biology or medical industry, I think you know, like science of like, hey, there are these experiments, there repeatedly do.
00:48:26.079 --> 00:48:42.480
I guess I don't want to kind of say too much out of distribution because I'm not a biology person, but I would just be really about understanding more there and understanding like so many of truly realistically long horizon.
00:48:42.480 --> 00:48:42.719
Right.
00:48:42.719 --> 00:48:47.920
what's the blocker there for why we can't deploy an agent, you easily operationalizable.
00:48:47.920 --> 00:48:54.320
What do we still need to solve to make it then be able to do those things?
00:48:54.320 --> 00:48:54.559
Yeah, yeah.
00:48:54.559 --> 00:48:55.199
Yeah.
00:48:55.440 --> 00:48:55.440
Two two reactions.
00:48:55.440 --> 00:48:59.440
One, I think one thing I talked to Alex about also on the last one on Terminal-Bench and harbor.
00:48:59.440 --> 00:49:06.239
I think the bet on the CLI, the bet on code in general as a an awesome one, right?
00:49:06.239 --> 00:49:16.719
Because to your point, it's underlying a lot of the affordances that agents are using inherently to interact with the actually move it.
00:49:16.719 --> 00:49:25.199
two, I 100% agree with this higher level notion that, hey, we into the benchmark building process, right?
00:49:25.199 --> 00:49:33.599
A lot of you know what we as scientists or engineers or have contact with are problems within our domains or ones, right?
00:49:33.599 --> 00:49:45.760
But there's a whole universe of you know use cases and and ways that we could be accelerating and you know taking think needs to be represented in the broader family of so lots of agreement there.
00:49:45.760 --> 00:49:47.518
I think the biomedical space is a super interesting one.
00:49:50.079 --> 00:49:50.079
Yeah, with you 100%.
00:49:51.760 --> 00:49:51.760
Five years out, does a benchmark still look like a
00:49:55.599 --> 00:50:04.559
If we asked this question around when SWE-bench was created, like not that different, you know, from like 2018 to like 2023, but I don't want to give I don't I don't think so.
00:50:04.559 --> 00:50:15.599
And I think like a lot of it has to do with kind of my my kind of that we discussed earlier, right?
00:50:15.599 --> 00:50:24.880
Which is that we're entering this realm of benchmarks and sort of technically are like borderline, or if not absolutely, possible for people to do.
00:50:24.880 --> 00:50:37.440
And I feel like a lot of benchmarking that has at least has been sort of like silos of you know what people can do.
00:50:37.440 --> 00:50:49.760
we did GPQA, we had question answering, we had sort of, you know, the T5 paper came out, uniting all those things.
00:50:49.760 --> 00:50:50.960
We had machine translation, blah, blah, blah.
00:50:50.960 --> 00:51:01.679
And these are sort of like different cuts of what people can do and so fundamentally, I think having tasks that are inspired by, machine do this?
00:51:01.679 --> 00:51:15.679
I think it was, I mean, it was a very, very good source of task And we kind of calibrated which benchmarks to pay attention to models were in terms of their performance, right?
00:51:15.679 --> 00:51:20.159
So I think SWE-bench in 2018 probably wouldn't have been that fairly timely.
00:51:20.159 --> 00:51:23.440
so going forwards, I am excited to see how it evolves.
00:51:23.440 --> 00:51:40.800
I just think like the human inspiration will be a lot, like the maybe is gonna be less and less the actual source of it, in my Yeah.
00:51:40.800 --> 00:51:49.280
And more so along like how do we engage with it, how do we work Yeah, being more of the core premise than just sort of like hill climbing what a person can do.
00:51:49.280 --> 00:52:02.719
and in terms of just like the form factor of it, you know, I feel environments, whether they're digital or maybe they become environments sort of in the coming years, I think that sentiment will stay around.
00:52:02.719 --> 00:52:17.599
I think the verification is where I'm kind of more curious how least in the regime of code, verification has evolved so much in the past three, four years.
00:52:17.599 --> 00:52:17.840
Yeah.
00:52:17.840 --> 00:52:18.239
And I don't expect that trend to stop at all.
00:52:21.280 --> 00:52:21.280
Last question for you.
00:52:21.280 --> 00:52:22.800
how can how can people leverage ProgramBench?
00:52:22.800 --> 00:52:26.639
Yeah, definitely.
00:52:27.840 --> 00:52:27.840
programbench.com, you know, the URLs there.
00:52:27.840 --> 00:52:32.320
It was a little pricey, but you know, we got it.
00:52:32.320 --> 00:52:35.039
so we're gonna set up a leaderboard very soon.
00:52:35.039 --> 00:52:42.639
We purposely kind of are encouraging people to just pick at we use mini-SWE-agent.
00:52:42.639 --> 00:52:45.840
You use your own scaffold if you think it's better, you know.
00:52:45.840 --> 00:53:01.199
single agent versus multi-agent scaffold, give that a I mean, as with SWE-bench, at some point I would be really excited to see people train their own models on this and see, like, hey, model?
00:53:01.199 --> 00:53:12.719
you know, well, is there really sort of, you know, given that what is training on that?
00:53:12.719 --> 00:53:19.039
You know, what how does that affect sort of how we train on code tasks when they get longer and sort of wider and bigger with larger solution spaces?
00:53:19.039 --> 00:53:20.000
So anyway, that's for setting up the leaderboard.
00:53:20.000 --> 00:53:53.280
I would also love to set up a way for people to add tasks because I think just ProgramBench as a formulation is meant to be very You know, there's very we want to impose as little constraints on You know, we just do executables, but there's no reason or an iOS application or a website or you know, anything that rendered software, you know.
00:53:53.280 --> 00:53:55.440
And so those would be, I think, the two call to actions.
00:53:55.440 --> 00:54:02.320
And, you know, Kilian and I are very sort of, you know, looking looking at the website traffic.
00:54:02.320 --> 00:54:14.559
So we're we're very looking forward to hopefully growing this just as excited as we are about sort of why this benchmark is
00:54:17.599 --> 00:54:17.599
Well, I'm excited as well.
00:54:17.599 --> 00:54:22.079
I hope you know, folks can contribute to the leaderboard, add and yeah, thanks again so much, John.
00:54:22.079 --> 00:54:24.800
This was an awesome chat.
00:54:25.920 --> 00:54:25.920
Yeah, this was fantastic.
00:54:25.920 --> 00:54:26.960
Thanks so much, thanks, thanks.