WEBVTT
00:00:00.080 --> 00:00:08.640
Okay, so a few days ago, an engineering manager at Anthropic posted that he's stopped giving coding interviews to the people he hires.
00:00:08.880 --> 00:00:10.240
His reason's pretty short.
00:00:10.480 --> 00:00:12.000
Claude passes the interviews.
00:00:12.080 --> 00:00:18.399
And if Claude can pass them, well, what good are they in determining whether one person is better than another?
00:00:18.640 --> 00:00:20.320
And he makes a really good point.
00:00:20.480 --> 00:00:26.160
Now, this is coming from Felix Riesberg, and he co-maintains Electron for Anthropic.
00:00:26.239 --> 00:00:29.760
That's the framework a whole lot of desktop apps are built on.
00:00:29.920 --> 00:00:36.640
Now, on October 27th, he posted on X quote, the coding interview is dead, at least for my team at Anthropic.
00:00:36.799 --> 00:00:40.880
And then he published an essay that included some advice for junior engineers.
00:00:41.039 --> 00:00:43.119
So what did he actually say?
00:00:43.439 --> 00:00:48.640
Felix shared how he used to give candidates small coding challenges in real code.
00:00:48.799 --> 00:00:58.799
One example he gives is a little React exercise, but he then goes on to say there's no more point in giving these React challenges than there is in quizzing people about assembly.
00:00:58.960 --> 00:01:00.560
Claude's now better at both.
00:01:00.719 --> 00:01:08.159
He describes these coding tests that he would give as things that were a weak but useful predictor of who would do well on his team.
00:01:08.400 --> 00:01:13.120
But then this month, he stopped using them because now they're not even useful.
00:01:13.280 --> 00:01:16.959
So what replaces it when he's evaluating younger engineers?
00:01:17.120 --> 00:01:18.319
Well, he doesn't really know yet.
00:01:18.480 --> 00:01:20.239
He says he hasn't gotten there quite yet.
00:01:20.319 --> 00:01:28.400
For now, he's back to reading resumes and portfolios, but the thing that he is looking for is small prototypes the person's built.
00:01:28.480 --> 00:01:37.519
And in those prototypes, the thing he especially calls out that he's evaluating is things like taste, empathy, and principled thinking.
00:01:37.760 --> 00:01:46.480
Now, really interesting that he didn't say an understanding of advanced AI models or the ability to churn out millions of lines of code in a given sprint.
00:01:46.560 --> 00:01:49.920
And yeah, I'm sure he's looking for people that are technically proficient.
00:01:50.000 --> 00:02:00.560
I suppose that's table stakes now, but the characteristics he's calling out, it almost sums up the job of a product manager, at least the role that we all want to have, right?
00:02:00.719 --> 00:02:07.439
Now, it's not the first time we've heard about this conundrum of sorts when it comes to coding challenges.
00:02:07.599 --> 00:02:10.479
Anthropic has been fighting this fight for a while.
00:02:10.560 --> 00:02:12.159
Um, there was Tristan Hume.
00:02:12.240 --> 00:02:17.280
Now, he leads performance optimization at Anthropic, and he wrote about this back in January too.
00:02:17.439 --> 00:02:23.360
He talked about how in November 2023 the team needed more performance engineers fast.
00:02:23.759 --> 00:02:29.759
So Tristan posted on Twitter and he got more candidates than the normal interview pipeline could even handle.
00:02:29.919 --> 00:02:32.560
And he spent two weeks designing a take-home test.
00:02:32.719 --> 00:02:38.319
He came up with a realistic problem and he asked engineers to optimize code for a simulated chip.
00:02:38.479 --> 00:02:44.800
Now, candidates could use AI on it, it wouldn't be cheating, because the whole point was to see how they worked with AI.
00:02:45.039 --> 00:02:50.800
More than a thousand people took it over 18 months, and it helped hire most of the current team he has.
00:02:50.960 --> 00:02:52.719
The test essentially worked.
00:02:52.879 --> 00:02:54.879
But then Claude got better.
00:02:55.120 --> 00:03:02.639
By May 2025, Claude 3.7 Sonnet was already beating more than half of the candidates when they handed the work to Claude Code.
00:03:02.879 --> 00:03:09.680
A pre-release Claude Opus 4 produced a better optimized solution than almost all humans in the four hours allowed.
00:03:09.840 --> 00:03:12.080
So Tristan redesigned that test.
00:03:12.240 --> 00:03:18.319
He found where Opus 4 started to struggle, made that the new starting point, and then cut the time limit to two hours.
00:03:18.479 --> 00:03:20.080
And that held for a few months.
00:03:20.240 --> 00:03:23.280
Then a pre-release Opus 4.5 came along.
00:03:23.439 --> 00:03:34.879
It ended up clearing the passing bar in under an hour, got stuck, and then once it was told what cycle count was possible, it found a trick and matched the best human score.
00:03:35.039 --> 00:03:38.400
Now, that human had heavily used Claude as well.
00:03:38.479 --> 00:03:39.759
I should point that out.
00:03:40.000 --> 00:03:43.280
But Tristan looked at two different paths from that point.
00:03:43.439 --> 00:03:49.439
He could either ban AI, but that was pretty hard to enforce, and people still need to work well with AI on the job.
00:03:49.520 --> 00:03:51.599
I mean, this is anthropic, right?
00:03:51.840 --> 00:03:56.159
Or he could raise the bar so a candidate had to beat Claude code on its own.
00:03:56.319 --> 00:03:59.360
But humans spend about half the time just reading the problem.
00:03:59.520 --> 00:04:04.400
So the best strategy would end up becoming just watching Claude work and learning from that.
00:04:04.560 --> 00:04:06.719
So he didn't really like either of those paths.
00:04:06.879 --> 00:04:08.960
Instead, he went with a third direction.
00:04:09.120 --> 00:04:17.360
The current test he now uses is a set of puzzles with a tiny, heavily constrained instruction set, far from anything in the model's training data.
00:04:17.600 --> 00:04:21.920
Opus 4.5 actually failed a medium to hard version of the test.
00:04:22.079 --> 00:04:25.360
And one design choice he made I thought was really interesting.
00:04:25.519 --> 00:04:27.839
There's no debugger, no visualizer.
00:04:28.079 --> 00:04:34.160
Candidates have to build their own tools because how someone invests in tooling tells you about their judgment.
00:04:34.399 --> 00:04:35.680
But he gave something up.
00:04:35.839 --> 00:04:39.199
He said the original test worked because it resembled real work.
00:04:39.439 --> 00:04:42.560
This new test works because it simulates novel work.
00:04:42.720 --> 00:04:44.480
And that's really interesting to think about.
00:04:44.639 --> 00:04:51.839
You could build a test that the model can't beat, but the harder you work at that, the further it ends up drifting from the real job.
00:04:52.079 --> 00:04:56.800
According to Tristan, designing an interview that represents the job is now harder than ever.
00:04:57.040 --> 00:05:00.079
Now let's go back to Felix, who we started with.
00:05:00.240 --> 00:05:05.439
Now, the decision he made, it's not like he was announcing some anthropic-wide policy.
00:05:05.519 --> 00:05:08.399
I mean, he's one manager deciding for his own team.
00:05:08.639 --> 00:05:13.279
But going back to his advice, he boiled things down to a few different things.
00:05:13.439 --> 00:05:18.800
First, he says it's important to focus on business impact instead of working the backlog.
00:05:18.959 --> 00:05:25.759
He says, pick one good idea, not five, and build a working prototype over a weekend instead of writing a PRD.
00:05:26.000 --> 00:05:27.839
Now, I love this.
00:05:28.000 --> 00:05:30.000
I love that it starts with business impact.
00:05:30.160 --> 00:05:34.240
I know my friend Matt LeMay, author of Impact First Product Teams, would love that too.
00:05:34.399 --> 00:05:36.879
Because let's face it, this is really important.
00:05:37.040 --> 00:05:40.879
The products we build are meant to create impact for our businesses.
00:05:41.040 --> 00:05:44.720
It's not just about building cool stuff as fast as possible.
00:05:44.959 --> 00:05:46.240
Yeah, speed matters.
00:05:46.319 --> 00:05:47.279
I'll get to that.
00:05:47.519 --> 00:05:50.160
But Felix gives a very specific example.
00:05:50.319 --> 00:05:55.199
There were two relatively new anthropic engineers, Anthony Morris and Alice Zhao.
00:05:55.439 --> 00:05:59.279
They built a graphical interface for Claude Code over a single weekend.
00:05:59.439 --> 00:06:03.040
The first public version shipped eight days later.
00:06:03.279 --> 00:06:08.639
So yes, speed matters, it matters a lot, but it's not the only thing.
00:06:08.879 --> 00:06:10.879
He also makes a point about fresh eyes.
00:06:11.040 --> 00:06:14.959
He said, newcomers don't know why something is supposedly impossible.
00:06:15.120 --> 00:06:17.040
So sometimes they try it anyway.
00:06:17.279 --> 00:06:18.399
And I love this.
00:06:18.560 --> 00:06:23.360
I've been saying it's so important for everybody, even those of us with a decade or two of product experience.
00:06:23.519 --> 00:06:32.079
In fact, maybe it's most important for those of us with a decade or two of product experience to be okay with being a beginner again.
00:06:32.240 --> 00:06:35.199
Sometimes it's what we know that sort of holds us back.
00:06:35.439 --> 00:06:40.240
Now, I know, again, this is coming from an engineering manager at Anthropic.
00:06:40.319 --> 00:06:43.199
And yes, Anthropic sells coding tools.
00:06:43.360 --> 00:06:50.639
So this whole story, right, where Claude is so good at coding that the interview broke, it's a flattering story for Anthropic.
00:06:50.800 --> 00:06:52.560
I do fully realize it.
00:06:52.720 --> 00:06:54.240
But it doesn't make him wrong.
00:06:54.480 --> 00:06:59.439
Maybe it is fair to take everything with a grain of salt here, but I actually think he's onto something.
00:06:59.600 --> 00:07:06.800
Now, there have been some critics to this whole concept of, like, hey, the coding tests, they're completely broken.
00:07:06.879 --> 00:07:13.199
I mean, some engineers have pointed out that nobody's found a better replacement for the 60-minute interview quite yet.
00:07:13.439 --> 00:07:16.240
Others have said, hey, look, anthropic is unusual.
00:07:16.399 --> 00:07:20.480
A lot of traditional companies will keep running algorithm tests for years.
00:07:20.720 --> 00:07:33.199
A few even pointed out the other side of the junior program that early career candidates, they already struggle to get past resume screening because some companies are reluctant to hire them at all, including Anthropic.
00:07:33.439 --> 00:07:41.120
You might remember just a couple weeks ago, the Anthropic Institute published a report on how employment is changing inside of Anthropic.
00:07:41.279 --> 00:07:51.199
And one finding, junior technical hires fell from 27% of the technical hires they made to 9% since early 2025.
00:07:51.279 --> 00:07:56.319
And the junior share of the technical staff dropped from 21% to 14%.
00:07:57.120 --> 00:08:10.079
Now, this essay tells junior engineers that hunger beats tenure, but the company's own numbers that they released, which we reported on not that long ago, they say that they're hiring far few juniors.
00:08:10.240 --> 00:08:14.639
Now, again, it doesn't necessarily mean that the information is conflicting.
00:08:14.800 --> 00:08:16.000
Both can be true.
00:08:16.160 --> 00:08:28.879
It can be harder for junior engineers to get in at Anthropic, and the way in can look more like a well-thought-out, impact-driven prototype that somebody builds rather than a coding test that they pass.
00:08:29.120 --> 00:08:36.960
But if you're a junior person reading Felix's advice, it is important that you know about this trend with junior engineers at Anthropic.
00:08:37.200 --> 00:08:41.039
Code commits per engineer were up five times from six months earlier.
00:08:41.200 --> 00:08:51.039
So the work of engineers is really starting to shift from coding product to overseeing the work to build the product that, of course, AI agents are doing.
00:08:51.279 --> 00:09:02.720
Now, one source from the research that the Anthropic Institute pointed out that we shared a couple weeks back is that the 20% of the work that was strategic is now 90% of the job.
00:09:02.879 --> 00:09:11.679
And if that's really true, you could see why a React exercise stopped telling Felix anything about whether somebody would be a good team member for him or not.
00:09:11.840 --> 00:09:13.919
Now, is Anthropic an outlier?
00:09:14.159 --> 00:09:16.080
Maybe not as much as you'd think.
00:09:16.320 --> 00:09:31.440
CoderPad recently did a survey of about 650 developers and recruiters, and they found that 34% ban AI completely, 46% allowed in some form, 20% decide case by case.
00:09:31.679 --> 00:09:39.759
Now, the company Canva, they require candidates to use AI tools like Copilot, Cursor, and Claude in technical interviews.
00:09:39.919 --> 00:09:47.039
What they saw is the strongest candidates controlled the boundaries of the problem and reviewed the AI's code with a critical eye.
00:09:47.279 --> 00:09:51.279
The weaker candidates struggled to judge what the AI actually handed them.
00:09:51.519 --> 00:09:59.360
So companies today are starting to not necessarily measure whether you can code, but whether you can tell if the code is any good.
00:09:59.679 --> 00:10:02.240
But what about product management specifically?
00:10:02.480 --> 00:10:04.799
Well, lots of companies are doing different things.
00:10:05.039 --> 00:10:11.440
LinkedIn recently renamed its entry-level APM program to Associate Product Builder.
00:10:11.600 --> 00:10:13.440
They remove the resume upload.
00:10:13.519 --> 00:10:20.960
They now ask applicants for a 60-second video of a product that people use, plus answers to six questions.
00:10:21.120 --> 00:10:24.639
Now, that's just one company and an example of how they've evolved.
00:10:24.720 --> 00:10:31.200
I know lots of companies, in fact, probably most companies, are still asking for a resume at the very least, right?
00:10:31.360 --> 00:10:34.639
But now it's no longer the only important piece that companies are looking at.
00:10:34.720 --> 00:10:39.759
They want to see what you can actually build and whether what you've built actually matters.
00:10:40.000 --> 00:10:46.240
Now, there is another potential problem to this whole, you know, the prototype is the new resume uh concept.
00:10:46.480 --> 00:10:53.440
The same AI that broke the coding test also makes a polished prototype very easy to produce, right?
00:10:53.759 --> 00:10:56.399
A slick demo is pretty cheap to build now.
00:10:56.559 --> 00:10:58.480
So what is it really a signal of?
00:10:58.720 --> 00:11:09.759
It shows that someone can build something, but it doesn't necessarily show how they build the right thing, or if they're willing to kill the wrong idea, or how can they actually work with people along the way?
00:11:10.000 --> 00:11:13.759
Also, demos without a scoring rubric, they introduce bias.
00:11:13.919 --> 00:11:17.519
I mean, measuring things like taste and judgment, it's pretty subjective.
00:11:17.679 --> 00:11:23.840
Felix says he's looking for taste, and I believe him, but taste is the hardest thing in the world to score for right now.
00:11:24.080 --> 00:11:26.879
So, what does this all mean for you?
00:11:27.200 --> 00:11:31.440
So think about the case study that you give candidates in the interview process.
00:11:31.679 --> 00:11:37.360
The product sense question, the improve this app prompt, the take home where they write a one-pager.
00:11:37.679 --> 00:11:48.639
I mean, if you'd happily advance a candidate who passed your test using good old Claude or ChatGPT or whatever they want to use, then is that round actually measuring a person anymore?
00:11:48.799 --> 00:11:52.000
Or is it measuring access that they have to a chat bot?
00:11:52.159 --> 00:11:56.399
I mean, all these tests, you've got to ask yourself, what was the point of the test anyway?
00:11:56.559 --> 00:11:58.159
What were you really trying to measure?
00:11:58.320 --> 00:12:04.480
Were you trying to understand whether a person can take a messy problem, make progress, and tell good work from bad?
00:12:04.720 --> 00:12:08.720
Maybe you should take a look at how your test is actually laid out right now.
00:12:08.879 --> 00:12:12.559
And maybe let your candidates use AI, but watch what they do with it.
00:12:12.799 --> 00:12:14.799
Do they take the first answer that they get?
00:12:14.879 --> 00:12:16.240
And is that obvious to you?
00:12:16.480 --> 00:12:18.960
Can you tell that they pushed back on the AI?
00:12:19.200 --> 00:12:22.799
Does their answer seem similar to all the other answers you're getting from people?
00:12:23.039 --> 00:12:26.080
Or do they notice what's missing from the user research?
00:12:26.240 --> 00:12:29.919
Can you find examples of them being different and with good reason?
00:12:30.159 --> 00:12:32.159
That's judgment showing up here.
00:12:32.240 --> 00:12:36.399
And it shows up in how someone reacts, which is harder to fake in a document.
00:12:36.639 --> 00:12:46.720
Now, one format that I keep seeing recommended is what people call an audit interview, where you hand the candidate a flawed piece of AI-generated work and ask them to review it.
00:12:46.879 --> 00:12:50.240
Now, for engineers, it's a repo with a confident bug in it.
00:12:50.399 --> 00:12:52.879
For product people, try to build the equivalent.
00:12:53.039 --> 00:13:00.320
Maybe it's a metrics plan that tracks the wrong thing or a launch plan that skips the one team that has to say yes.
00:13:00.720 --> 00:13:04.159
Watch what they catch, what they miss, how they explain it to you.
00:13:04.320 --> 00:13:08.960
You may end up learning more in 20 minutes than you did from your last three case studies.
00:13:09.279 --> 00:13:11.120
But set a rubric first.
00:13:11.279 --> 00:13:18.399
If you can't write down what a good answer looks like in advance, you could end up hiring the person that sounds the most like you.
00:13:18.720 --> 00:13:22.080
That's the bias I was talking about before, and you've got to watch for that.
00:13:22.240 --> 00:13:24.080
And look at your job postings too.
00:13:24.240 --> 00:13:27.519
I mean, Felix's whole argument is that hunger beats tenure.
00:13:27.679 --> 00:13:29.519
Well, go on and read your own listings.
00:13:29.679 --> 00:13:34.000
Count how many times it asks for five years or 10 years of experience.
00:13:34.240 --> 00:13:46.159
If the thing you actually want is somebody who builds a working version of an idea and has the taste to know whether it's any good, years on a resume might not be a great proxy for that anymore.
00:13:46.320 --> 00:13:49.279
I'm not saying throw out experience altogether.
00:13:49.440 --> 00:13:52.559
Like experienced product managers, they have scarred tissue.
00:13:52.720 --> 00:13:59.200
Scarred tissue could be an indicator of real judgment, but don't overlook that hungry junior either.
00:13:59.360 --> 00:14:08.159
And hey, if a company like Anthropic is hiring fewer juniors, maybe it's a competitive advantage for your team when you're actually doing the hiring.
00:14:08.399 --> 00:14:11.600
So here's one thing I would do this week.
00:14:11.840 --> 00:14:16.960
Pick one exercise from your PM interview loop and run it through Claude Cold.
00:14:17.200 --> 00:14:18.080
No coaching.
00:14:18.240 --> 00:14:25.279
Just sort of read what comes back and then ask yourself if I got this from a person, would I have hired this person?
00:14:25.519 --> 00:14:31.759
If the answer is yes, I think it's time to start changing your exercise right now.
00:14:32.000 --> 00:14:35.200
All right, that's gonna be a wrap on this week's episode.
00:14:35.440 --> 00:14:42.879
If you found this episode valuable, please subscribe, tell a friend, and if you have any suggestions for me, leave a comment below.
00:14:42.960 --> 00:14:44.960
I promise you, I read all of the comments.
00:14:45.120 --> 00:14:47.600
I will even respond to your comments when I see them.
00:14:47.679 --> 00:14:50.399
So anyway, I will see you right here next week.
00:14:50.480 --> 00:14:57.120
Once again, my name is Mike Belsito, and brought to you by the team at Mind the Product, this is now shipping.
00:00:00.080 --> 00:00:08.640
Okay, so a few days ago, an engineering manager at Anthropic posted that he's stopped giving coding interviews to the people he hires.
00:00:08.880 --> 00:00:10.240
His reason's pretty short.
00:00:10.480 --> 00:00:12.000
Claude passes the interviews.
00:00:12.080 --> 00:00:18.399
And if Claude can pass them, well, what good are they in determining whether one person is better than another?
00:00:18.640 --> 00:00:20.320
And he makes a really good point.
00:00:20.480 --> 00:00:26.160
Now, this is coming from Felix Riesberg, and he co-maintains Electron for Anthropic.
00:00:26.239 --> 00:00:29.760
That's the framework a whole lot of desktop apps are built on.
00:00:29.920 --> 00:00:36.640
Now, on October 27th, he posted on X quote, the coding interview is dead, at least for my team at Anthropic.
00:00:36.799 --> 00:00:40.880
And then he published an essay that included some advice for junior engineers.
00:00:41.039 --> 00:00:43.119
So what did he actually say?
00:00:43.439 --> 00:00:48.640
Felix shared how he used to give candidates small coding challenges in real code.
00:00:48.799 --> 00:00:58.799
One example he gives is a little React exercise, but he then goes on to say there's no more point in giving these React challenges than there is in quizzing people about assembly.
00:00:58.960 --> 00:01:00.560
Claude's now better at both.
00:01:00.719 --> 00:01:08.159
He describes these coding tests that he would give as things that were a weak but useful predictor of who would do well on his team.
00:01:08.400 --> 00:01:13.120
But then this month, he stopped using them because now they're not even useful.
00:01:13.280 --> 00:01:16.959
So what replaces it when he's evaluating younger engineers?
00:01:17.120 --> 00:01:18.319
Well, he doesn't really know yet.
00:01:18.480 --> 00:01:20.239
He says he hasn't gotten there quite yet.
00:01:20.319 --> 00:01:28.400
For now, he's back to reading resumes and portfolios, but the thing that he is looking for is small prototypes the person's built.
00:01:28.480 --> 00:01:37.519
And in those prototypes, the thing he especially calls out that he's evaluating is things like taste, empathy, and principled thinking.
00:01:37.760 --> 00:01:46.480
Now, really interesting that he didn't say an understanding of advanced AI models or the ability to churn out millions of lines of code in a given sprint.
00:01:46.560 --> 00:01:49.920
And yeah, I'm sure he's looking for people that are technically proficient.
00:01:50.000 --> 00:02:00.560
I suppose that's table stakes now, but the characteristics he's calling out, it almost sums up the job of a product manager, at least the role that we all want to have, right?
00:02:00.719 --> 00:02:07.439
Now, it's not the first time we've heard about this conundrum of sorts when it comes to coding challenges.
00:02:07.599 --> 00:02:10.479
Anthropic has been fighting this fight for a while.
00:02:10.560 --> 00:02:12.159
Um, there was Tristan Hume.
00:02:12.240 --> 00:02:17.280
Now, he leads performance optimization at Anthropic, and he wrote about this back in January too.
00:02:17.439 --> 00:02:23.360
He talked about how in November 2023 the team needed more performance engineers fast.
00:02:23.759 --> 00:02:29.759
So Tristan posted on Twitter and he got more candidates than the normal interview pipeline could even handle.
00:02:29.919 --> 00:02:32.560
And he spent two weeks designing a take-home test.
00:02:32.719 --> 00:02:38.319
He came up with a realistic problem and he asked engineers to optimize code for a simulated chip.
00:02:38.479 --> 00:02:44.800
Now, candidates could use AI on it, it wouldn't be cheating, because the whole point was to see how they worked with AI.
00:02:45.039 --> 00:02:50.800
More than a thousand people took it over 18 months, and it helped hire most of the current team he has.
00:02:50.960 --> 00:02:52.719
The test essentially worked.
00:02:52.879 --> 00:02:54.879
But then Claude got better.
00:02:55.120 --> 00:03:02.639
By May 2025, Claude 3.7 Sonnet was already beating more than half of the candidates when they handed the work to Claude Code.
00:03:02.879 --> 00:03:09.680
A pre-release Claude Opus 4 produced a better optimized solution than almost all humans in the four hours allowed.
00:03:09.840 --> 00:03:12.080
So Tristan redesigned that test.
00:03:12.240 --> 00:03:18.319
He found where Opus 4 started to struggle, made that the new starting point, and then cut the time limit to two hours.
00:03:18.479 --> 00:03:20.080
And that held for a few months.
00:03:20.240 --> 00:03:23.280
Then a pre-release Opus 4.5 came along.
00:03:23.439 --> 00:03:34.879
It ended up clearing the passing bar in under an hour, got stuck, and then once it was told what cycle count was possible, it found a trick and matched the best human score.
00:03:35.039 --> 00:03:38.400
Now, that human had heavily used Claude as well.
00:03:38.479 --> 00:03:39.759
I should point that out.
00:03:40.000 --> 00:03:43.280
But Tristan looked at two different paths from that point.
00:03:43.439 --> 00:03:49.439
He could either ban AI, but that was pretty hard to enforce, and people still need to work well with AI on the job.
00:03:49.520 --> 00:03:51.599
I mean, this is anthropic, right?
00:03:51.840 --> 00:03:56.159
Or he could raise the bar so a candidate had to beat Claude code on its own.
00:03:56.319 --> 00:03:59.360
But humans spend about half the time just reading the problem.
00:03:59.520 --> 00:04:04.400
So the best strategy would end up becoming just watching Claude work and learning from that.
00:04:04.560 --> 00:04:06.719
So he didn't really like either of those paths.
00:04:06.879 --> 00:04:08.960
Instead, he went with a third direction.
00:04:09.120 --> 00:04:17.360
The current test he now uses is a set of puzzles with a tiny, heavily constrained instruction set, far from anything in the model's training data.
00:04:17.600 --> 00:04:21.920
Opus 4.5 actually failed a medium to hard version of the test.
00:04:22.079 --> 00:04:25.360
And one design choice he made I thought was really interesting.
00:04:25.519 --> 00:04:27.839
There's no debugger, no visualizer.
00:04:28.079 --> 00:04:34.160
Candidates have to build their own tools because how someone invests in tooling tells you about their judgment.
00:04:34.399 --> 00:04:35.680
But he gave something up.
00:04:35.839 --> 00:04:39.199
He said the original test worked because it resembled real work.
00:04:39.439 --> 00:04:42.560
This new test works because it simulates novel work.
00:04:42.720 --> 00:04:44.480
And that's really interesting to think about.
00:04:44.639 --> 00:04:51.839
You could build a test that the model can't beat, but the harder you work at that, the further it ends up drifting from the real job.
00:04:52.079 --> 00:04:56.800
According to Tristan, designing an interview that represents the job is now harder than ever.
00:04:57.040 --> 00:05:00.079
Now let's go back to Felix, who we started with.
00:05:00.240 --> 00:05:05.439
Now, the decision he made, it's not like he was announcing some anthropic-wide policy.
00:05:05.519 --> 00:05:08.399
I mean, he's one manager deciding for his own team.
00:05:08.639 --> 00:05:13.279
But going back to his advice, he boiled things down to a few different things.
00:05:13.439 --> 00:05:18.800
First, he says it's important to focus on business impact instead of working the backlog.
00:05:18.959 --> 00:05:25.759
He says, pick one good idea, not five, and build a working prototype over a weekend instead of writing a PRD.
00:05:26.000 --> 00:05:27.839
Now, I love this.
00:05:28.000 --> 00:05:30.000
I love that it starts with business impact.
00:05:30.160 --> 00:05:34.240
I know my friend Matt LeMay, author of Impact First Product Teams, would love that too.
00:05:34.399 --> 00:05:36.879
Because let's face it, this is really important.
00:05:37.040 --> 00:05:40.879
The products we build are meant to create impact for our businesses.
00:05:41.040 --> 00:05:44.720
It's not just about building cool stuff as fast as possible.
00:05:44.959 --> 00:05:46.240
Yeah, speed matters.
00:05:46.319 --> 00:05:47.279
I'll get to that.
00:05:47.519 --> 00:05:50.160
But Felix gives a very specific example.
00:05:50.319 --> 00:05:55.199
There were two relatively new anthropic engineers, Anthony Morris and Alice Zhao.
00:05:55.439 --> 00:05:59.279
They built a graphical interface for Claude Code over a single weekend.
00:05:59.439 --> 00:06:03.040
The first public version shipped eight days later.
00:06:03.279 --> 00:06:08.639
So yes, speed matters, it matters a lot, but it's not the only thing.
00:06:08.879 --> 00:06:10.879
He also makes a point about fresh eyes.
00:06:11.040 --> 00:06:14.959
He said, newcomers don't know why something is supposedly impossible.
00:06:15.120 --> 00:06:17.040
So sometimes they try it anyway.
00:06:17.279 --> 00:06:18.399
And I love this.
00:06:18.560 --> 00:06:23.360
I've been saying it's so important for everybody, even those of us with a decade or two of product experience.
00:06:23.519 --> 00:06:32.079
In fact, maybe it's most important for those of us with a decade or two of product experience to be okay with being a beginner again.
00:06:32.240 --> 00:06:35.199
Sometimes it's what we know that sort of holds us back.
00:06:35.439 --> 00:06:40.240
Now, I know, again, this is coming from an engineering manager at Anthropic.
00:06:40.319 --> 00:06:43.199
And yes, Anthropic sells coding tools.
00:06:43.360 --> 00:06:50.639
So this whole story, right, where Claude is so good at coding that the interview broke, it's a flattering story for Anthropic.
00:06:50.800 --> 00:06:52.560
I do fully realize it.
00:06:52.720 --> 00:06:54.240
But it doesn't make him wrong.
00:06:54.480 --> 00:06:59.439
Maybe it is fair to take everything with a grain of salt here, but I actually think he's onto something.
00:06:59.600 --> 00:07:06.800
Now, there have been some critics to this whole concept of, like, hey, the coding tests, they're completely broken.
00:07:06.879 --> 00:07:13.199
I mean, some engineers have pointed out that nobody's found a better replacement for the 60-minute interview quite yet.
00:07:13.439 --> 00:07:16.240
Others have said, hey, look, anthropic is unusual.
00:07:16.399 --> 00:07:20.480
A lot of traditional companies will keep running algorithm tests for years.
00:07:20.720 --> 00:07:33.199
A few even pointed out the other side of the junior program that early career candidates, they already struggle to get past resume screening because some companies are reluctant to hire them at all, including Anthropic.
00:07:33.439 --> 00:07:41.120
You might remember just a couple weeks ago, the Anthropic Institute published a report on how employment is changing inside of Anthropic.
00:07:41.279 --> 00:07:51.199
And one finding, junior technical hires fell from 27% of the technical hires they made to 9% since early 2025.
00:07:51.279 --> 00:07:56.319
And the junior share of the technical staff dropped from 21% to 14%.
00:07:57.120 --> 00:08:10.079
Now, this essay tells junior engineers that hunger beats tenure, but the company's own numbers that they released, which we reported on not that long ago, they say that they're hiring far few juniors.
00:08:10.240 --> 00:08:14.639
Now, again, it doesn't necessarily mean that the information is conflicting.
00:08:14.800 --> 00:08:16.000
Both can be true.
00:08:16.160 --> 00:08:28.879
It can be harder for junior engineers to get in at Anthropic, and the way in can look more like a well-thought-out, impact-driven prototype that somebody builds rather than a coding test that they pass.
00:08:29.120 --> 00:08:36.960
But if you're a junior person reading Felix's advice, it is important that you know about this trend with junior engineers at Anthropic.
00:08:37.200 --> 00:08:41.039
Code commits per engineer were up five times from six months earlier.
00:08:41.200 --> 00:08:51.039
So the work of engineers is really starting to shift from coding product to overseeing the work to build the product that, of course, AI agents are doing.
00:08:51.279 --> 00:09:02.720
Now, one source from the research that the Anthropic Institute pointed out that we shared a couple weeks back is that the 20% of the work that was strategic is now 90% of the job.
00:09:02.879 --> 00:09:11.679
And if that's really true, you could see why a React exercise stopped telling Felix anything about whether somebody would be a good team member for him or not.
00:09:11.840 --> 00:09:13.919
Now, is Anthropic an outlier?
00:09:14.159 --> 00:09:16.080
Maybe not as much as you'd think.
00:09:16.320 --> 00:09:31.440
CoderPad recently did a survey of about 650 developers and recruiters, and they found that 34% ban AI completely, 46% allowed in some form, 20% decide case by case.
00:09:31.679 --> 00:09:39.759
Now, the company Canva, they require candidates to use AI tools like Copilot, Cursor, and Claude in technical interviews.
00:09:39.919 --> 00:09:47.039
What they saw is the strongest candidates controlled the boundaries of the problem and reviewed the AI's code with a critical eye.
00:09:47.279 --> 00:09:51.279
The weaker candidates struggled to judge what the AI actually handed them.
00:09:51.519 --> 00:09:59.360
So companies today are starting to not necessarily measure whether you can code, but whether you can tell if the code is any good.
00:09:59.679 --> 00:10:02.240
But what about product management specifically?
00:10:02.480 --> 00:10:04.799
Well, lots of companies are doing different things.
00:10:05.039 --> 00:10:11.440
LinkedIn recently renamed its entry-level APM program to Associate Product Builder.
00:10:11.600 --> 00:10:13.440
They remove the resume upload.
00:10:13.519 --> 00:10:20.960
They now ask applicants for a 60-second video of a product that people use, plus answers to six questions.
00:10:21.120 --> 00:10:24.639
Now, that's just one company and an example of how they've evolved.
00:10:24.720 --> 00:10:31.200
I know lots of companies, in fact, probably most companies, are still asking for a resume at the very least, right?
00:10:31.360 --> 00:10:34.639
But now it's no longer the only important piece that companies are looking at.
00:10:34.720 --> 00:10:39.759
They want to see what you can actually build and whether what you've built actually matters.
00:10:40.000 --> 00:10:46.240
Now, there is another potential problem to this whole, you know, the prototype is the new resume uh concept.
00:10:46.480 --> 00:10:53.440
The same AI that broke the coding test also makes a polished prototype very easy to produce, right?
00:10:53.759 --> 00:10:56.399
A slick demo is pretty cheap to build now.
00:10:56.559 --> 00:10:58.480
So what is it really a signal of?
00:10:58.720 --> 00:11:09.759
It shows that someone can build something, but it doesn't necessarily show how they build the right thing, or if they're willing to kill the wrong idea, or how can they actually work with people along the way?
00:11:10.000 --> 00:11:13.759
Also, demos without a scoring rubric, they introduce bias.
00:11:13.919 --> 00:11:17.519
I mean, measuring things like taste and judgment, it's pretty subjective.
00:11:17.679 --> 00:11:23.840
Felix says he's looking for taste, and I believe him, but taste is the hardest thing in the world to score for right now.
00:11:24.080 --> 00:11:26.879
So, what does this all mean for you?
00:11:27.200 --> 00:11:31.440
So think about the case study that you give candidates in the interview process.
00:11:31.679 --> 00:11:37.360
The product sense question, the improve this app prompt, the take home where they write a one-pager.
00:11:37.679 --> 00:11:48.639
I mean, if you'd happily advance a candidate who passed your test using good old Claude or ChatGPT or whatever they want to use, then is that round actually measuring a person anymore?
00:11:48.799 --> 00:11:52.000
Or is it measuring access that they have to a chat bot?
00:11:52.159 --> 00:11:56.399
I mean, all these tests, you've got to ask yourself, what was the point of the test anyway?
00:11:56.559 --> 00:11:58.159
What were you really trying to measure?
00:11:58.320 --> 00:12:04.480
Were you trying to understand whether a person can take a messy problem, make progress, and tell good work from bad?
00:12:04.720 --> 00:12:08.720
Maybe you should take a look at how your test is actually laid out right now.
00:12:08.879 --> 00:12:12.559
And maybe let your candidates use AI, but watch what they do with it.
00:12:12.799 --> 00:12:14.799
Do they take the first answer that they get?
00:12:14.879 --> 00:12:16.240
And is that obvious to you?
00:12:16.480 --> 00:12:18.960
Can you tell that they pushed back on the AI?
00:12:19.200 --> 00:12:22.799
Does their answer seem similar to all the other answers you're getting from people?
00:12:23.039 --> 00:12:26.080
Or do they notice what's missing from the user research?
00:12:26.240 --> 00:12:29.919
Can you find examples of them being different and with good reason?
00:12:30.159 --> 00:12:32.159
That's judgment showing up here.
00:12:32.240 --> 00:12:36.399
And it shows up in how someone reacts, which is harder to fake in a document.
00:12:36.639 --> 00:12:46.720
Now, one format that I keep seeing recommended is what people call an audit interview, where you hand the candidate a flawed piece of AI-generated work and ask them to review it.
00:12:46.879 --> 00:12:50.240
Now, for engineers, it's a repo with a confident bug in it.
00:12:50.399 --> 00:12:52.879
For product people, try to build the equivalent.
00:12:53.039 --> 00:13:00.320
Maybe it's a metrics plan that tracks the wrong thing or a launch plan that skips the one team that has to say yes.
00:13:00.720 --> 00:13:04.159
Watch what they catch, what they miss, how they explain it to you.
00:13:04.320 --> 00:13:08.960
You may end up learning more in 20 minutes than you did from your last three case studies.
00:13:09.279 --> 00:13:11.120
But set a rubric first.
00:13:11.279 --> 00:13:18.399
If you can't write down what a good answer looks like in advance, you could end up hiring the person that sounds the most like you.
00:13:18.720 --> 00:13:22.080
That's the bias I was talking about before, and you've got to watch for that.
00:13:22.240 --> 00:13:24.080
And look at your job postings too.
00:13:24.240 --> 00:13:27.519
I mean, Felix's whole argument is that hunger beats tenure.
00:13:27.679 --> 00:13:29.519
Well, go on and read your own listings.
00:13:29.679 --> 00:13:34.000
Count how many times it asks for five years or 10 years of experience.
00:13:34.240 --> 00:13:46.159
If the thing you actually want is somebody who builds a working version of an idea and has the taste to know whether it's any good, years on a resume might not be a great proxy for that anymore.
00:13:46.320 --> 00:13:49.279
I'm not saying throw out experience altogether.
00:13:49.440 --> 00:13:52.559
Like experienced product managers, they have scarred tissue.
00:13:52.720 --> 00:13:59.200
Scarred tissue could be an indicator of real judgment, but don't overlook that hungry junior either.
00:13:59.360 --> 00:14:08.159
And hey, if a company like Anthropic is hiring fewer juniors, maybe it's a competitive advantage for your team when you're actually doing the hiring.
00:14:08.399 --> 00:14:11.600
So here's one thing I would do this week.
00:14:11.840 --> 00:14:16.960
Pick one exercise from your PM interview loop and run it through Claude Cold.
00:14:17.200 --> 00:14:18.080
No coaching.
00:14:18.240 --> 00:14:25.279
Just sort of read what comes back and then ask yourself if I got this from a person, would I have hired this person?
00:14:25.519 --> 00:14:31.759
If the answer is yes, I think it's time to start changing your exercise right now.
00:14:32.000 --> 00:14:35.200
All right, that's gonna be a wrap on this week's episode.
00:14:35.440 --> 00:14:42.879
If you found this episode valuable, please subscribe, tell a friend, and if you have any suggestions for me, leave a comment below.
00:14:42.960 --> 00:14:44.960
I promise you, I read all of the comments.
00:14:45.120 --> 00:14:47.600
I will even respond to your comments when I see them.
00:14:47.679 --> 00:14:50.399
So anyway, I will see you right here next week.
00:14:50.480 --> 00:14:57.120
Once again, my name is Mike Belsito, and brought to you by the team at Mind the Product, this is now shipping.
