TRANSKRIPTIO 🔗
WEBVTT
00:00:00.160 --> 00:00:04.799
I just feel like this like very textual, simple interface isn't cutting it.
00:00:04.879 --> 00:00:07.519
And I want agents in there proactively doing stuff.
00:00:07.919 --> 00:00:14.800
I do think agents prompting us more would actually help relieve our cognitive load, which a lot of us I think are struggling with in the age of agents.
00:00:14.880 --> 00:00:17.440
And you can't solve it with a multiple choice text question.
00:00:17.760 --> 00:00:20.000
You're asking me to pick between a bunch of MPM libraries.
00:00:20.160 --> 00:00:22.640
Give me like a comprehensive report of how these compare.
00:00:22.800 --> 00:00:24.640
Don't wait for me to prompt you.
00:00:25.839 --> 00:00:32.479
The AI Native Dev is a podcast with developers and engineering leads at the cutting edge of AI and agentic coding.
00:00:32.640 --> 00:00:42.799
Join your host, Guy Fajani, and meet Simon Maple every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today.
00:00:43.359 --> 00:00:45.920
This is the AI Native Dev.
00:00:48.479 --> 00:00:50.560
Yesterday, you were writing code.
00:00:50.719 --> 00:00:52.640
Today, you're prompting agents.
00:00:52.880 --> 00:00:55.359
Tomorrow, you'll be running a software factory.
00:00:55.679 --> 00:00:58.960
Identific code is moving fast enough to give anyone with that.
00:00:59.439 --> 00:01:02.159
This June, AI DevCon London sold out.
00:01:02.479 --> 00:01:09.599
Hundreds of developers, engineering leads, and VPs hacked in to hear speakers from our topic, OpenAI, Netflix, and GitHub.
00:01:09.920 --> 00:01:12.799
Cut through the hype and get to what actually works.
00:01:13.040 --> 00:01:15.680
This November, we're bringing it back to New York.
00:01:15.920 --> 00:01:20.239
Big up, folder, and full of people actually building the agency future.
00:01:20.480 --> 00:01:29.920
Three days of real talks, real tools, real answers, whether you're directing the agents, building the loops themselves, leading the teams, or setting the strategy.
00:01:30.239 --> 00:01:36.079
As a podcast listener, you can get 15% off your ticket using promo code pod 50.
00:01:37.200 --> 00:01:41.280
We have so much planned, and we can't wait to have you join us in New York.
00:01:45.840 --> 00:01:49.599
Hello and welcome to another episode of the AI Native Dev.
00:01:49.760 --> 00:02:06.000
Joining me today is Maggie Appleton, a staff research engineer at GitHub Next, and we're going to have a great conversation tackling such topics as how humans and agents interact and what are the handoff points between them when we think about planning, when we think about development, and so on.
00:02:06.079 --> 00:02:09.520
So we've got a huge number of topics and depth that we're going to talk about.
00:02:09.759 --> 00:02:11.280
Maggie, welcome to the podcast.
00:02:11.360 --> 00:02:11.919
How are you doing?
00:02:12.159 --> 00:02:12.719
Good, thank you.
00:02:12.800 -->
Load full transcript
00:02:13.360
Thank you for having me.
00:02:13.599 --> 00:02:14.319
Oh, absolute pleasure.
00:02:14.400 --> 00:02:18.400
Now, Maggie, tell us a little bit about uh your role and what you do at GitHub next.
00:02:18.719 --> 00:02:18.960
Sure.
00:02:19.039 --> 00:02:21.039
So technically, staff research engineer.
00:02:21.360 --> 00:02:23.120
That is the title of almost everyone on the team.
00:02:23.280 --> 00:02:27.439
I'm actually a designer, I'm a front-end engineer as well, but I'm not the best.
00:02:27.520 --> 00:02:32.000
I'm much more at home making like uh exploratory interfaces, visual design.
00:02:32.080 --> 00:02:38.639
I came from a background that was like an illustrator art director, and then moved into more front-end stuff working on like dev education.
00:02:38.800 --> 00:02:51.280
Um, and then I I just love a lot of things like uh theory of programming and like researching it, what's the future of software development, which is why GitHub Next is a very good place for me to just kind of do interesting experiments about what GitHub should work on next.
00:02:51.520 --> 00:02:59.840
Yeah, no uh I was just about to say what is GitHub Next, but for people for people who haven't heard of it, tell us a little bit about the kind of it's like a lab research space, right?
00:02:59.919 --> 00:03:00.719
It's super interesting.
00:03:00.879 --> 00:03:06.080
Tell us a little bit about a few of the interesting projects that are being worked on and that have come out of GitHub Next.
00:03:06.319 --> 00:03:06.400
Sure.
00:03:06.479 --> 00:03:08.879
So yeah, GitHub Next is is the lab's RD branch.
00:03:09.039 --> 00:03:11.039
So like pretty standard like innovation.
00:03:11.199 --> 00:03:16.879
Uh what the usual product engineering org wouldn't work on because it's too risky or too far out there.
00:03:17.039 --> 00:03:20.879
This lab works on like big swings, uh further in the future sort of stuff.
00:03:20.960 --> 00:03:27.680
Uh and this lab originally made the first GitHub co-pilots, like the first ever AI coding agent before anyone else had made one.
00:03:27.840 --> 00:03:30.479
Um and of course now there are plenty, but this was like the OG.
00:03:30.560 --> 00:03:33.039
I wasn't on the team, so I'm not taking credit for that.
00:03:33.520 --> 00:03:36.319
But uh there's tons of always tons of interesting projects.
00:03:36.400 --> 00:03:38.479
I feel like we're in an exploratory phase right now.
00:03:38.639 --> 00:03:40.319
So everyone's prototyping like mad.
00:03:40.400 --> 00:03:46.719
Every morning I wake up to like five new prototypes to try to understand and figure out if they have legs, like a lot of stuff's early.
00:03:46.800 --> 00:03:58.240
Um but some of the big projects out of the lab at the moment is like um agentic workflows you had Don on a little while ago to talk about, essentially software factories, or like he's trying to lobby for the phrase software farms, which I like more.
00:03:58.400 --> 00:04:00.639
Like you're harvesting the crops of agents all the time.
00:04:00.719 --> 00:04:02.639
It's less mechanical, it's more like growing.
00:04:02.879 --> 00:04:03.039
Yes.
00:04:03.360 --> 00:04:11.439
Um, and then I'm working on a project called Chopin, which is all around how we plan with agents and collaboratively with other developers and other members of your team.
00:04:11.520 --> 00:04:25.600
So not just you and an agent alone, but you and your whole team planning more rigorously together and more visually, because I'm big on like diagrams, visual explanations, like tapping into how graphic design can help people understand complex concepts.
00:04:25.920 --> 00:04:27.439
So for me, that's a big piece of it.
00:04:27.519 --> 00:04:34.560
But other people working on the project are more excited about you know the multiplayer piece and and tapping into like letting all your local agents work with it.
00:04:34.639 --> 00:04:35.920
So there's there's layers to the project.
00:04:36.160 --> 00:04:39.759
I totally thought you were gonna say, like, Chopin, it was gonna be an orchestration piece or something like that.
00:04:39.839 --> 00:04:41.199
But I suppose with multiplayer it's a lot of people.
00:04:41.360 --> 00:04:42.079
There's some in there.
00:04:42.160 --> 00:04:43.360
There's a lot of stuff in there.
00:04:43.519 --> 00:04:47.360
Um my uh Shistov is the engineer on it, and he's from Poland.
00:04:47.439 --> 00:04:53.279
So Chopin's a very famous Polish um composer, so he loved the name because it's all it is about orchestrating agents.
00:04:53.360 --> 00:04:59.680
There's multiple agents in there, you can kind of hand off to implementation and see how the work flows from plan through to finished product.
00:04:59.759 --> 00:05:13.759
So there's a few things to it, but the piece I love is the like the collaborative planning, the visuals, the like rich, rich artifacts to help developers understand what they're about to ask agents to do, and then when the work's done, trying to comprehend what the agents did.
00:05:14.240 --> 00:05:16.480
The two big bottlenecks, I think, in in current work.
00:05:16.800 --> 00:05:17.040
Nice.
00:05:17.279 --> 00:05:19.360
So you're in the GitHub.next space.
00:05:19.439 --> 00:05:20.800
Is it GitHub.next or GitHub next?
00:05:21.040 --> 00:05:21.439
GitHub next.
00:05:21.600 --> 00:05:22.000
GitHub next.
00:05:22.240 --> 00:05:22.800
Like the next GitHub.
00:05:23.040 --> 00:05:23.519
The next GitHub.
00:05:23.759 --> 00:05:31.120
I presume, I presume you refer to to current, like you know, the mainstream GitHub is either GitHub old or GitHub current.
00:05:31.439 --> 00:05:31.519
Yeah.
00:05:31.759 --> 00:05:37.920
And then I in my head there's a GitHub uh prev, which is like legacy, working with legacy clients or something.
00:05:38.079 --> 00:05:40.000
If you say, if you say no, I'm just gonna imagine that anyway.
00:05:40.079 --> 00:05:41.680
So that's we kind of call it.com.
00:05:42.399 --> 00:05:42.639
Oh, okay.
00:05:42.879 --> 00:05:47.439
Because it's like we we don't tend to work on dot com because dot com is like a big, a big thing, right?
00:05:47.519 --> 00:05:49.839
With a lot of real clients needed to stay the same.
00:05:49.920 --> 00:05:50.079
Yeah.
00:05:50.240 --> 00:05:56.480
So we tend to work on things that are like more prototypes on the side or might end up rolling up into dot com, but not directly on the surface.
00:05:56.720 --> 00:05:57.519
Yeah, nice, nice.
00:05:57.680 --> 00:06:01.360
So, Maggie, one of the things I love is going through past comments that people have made.
00:06:01.759 --> 00:06:06.399
And and yeah, dragging up the past and and and and showing the dirty washing on the on show here.
00:06:06.560 --> 00:06:10.480
So one of your lines was nine women make a baby in one month.
00:06:10.639 --> 00:06:12.560
Uh and that's a great I love that logic.
00:06:12.639 --> 00:06:13.439
I love the line.
00:06:13.600 --> 00:06:18.879
Um so concretely, what does a what does a team look like that's you know shipping five features a day?
00:06:19.199 --> 00:06:19.439
Yeah.
00:06:19.519 --> 00:06:29.040
Uh so this line came out of a talk I did uh that was about a collaborative uh developer uh platform that we had made called Ace or Prototype that Chapan is kind of related to.
00:06:29.120 --> 00:06:36.959
And it was essentially just like Slack, but with um micro VMs for code sandboxes in real time and agents in there as well.
00:06:37.040 --> 00:06:39.279
So like all of you in one uh environment.
00:06:39.439 --> 00:06:47.199
And this is now like happening a lot more with their products now, like Buzz from Jack Dorsey, and like even Claude getting into Slack is like kind of going that direction.
00:06:47.439 --> 00:06:51.040
This was maybe like uh six months ago that we had a prototype of in this direction.
00:06:51.120 --> 00:06:53.680
So it's nice to see other people agree with our theory.
00:06:53.920 --> 00:07:01.920
Um, and the the problem we were trying to solve was that uh everyone's going really, really fast with their agents locally in their terminals or their local desktop apps.
00:07:02.079 --> 00:07:11.360
So everyone on your team is accelerated, but like scaling up one person doesn't lead to like good collaborative products because everyone's off on their own branches doing their own thing.
00:07:11.439 --> 00:07:18.480
It's really you get carry merge conflicts eventually, or you get features that like other people didn't agree to or didn't get to put input on.
00:07:18.720 --> 00:07:28.000
So one person could build something and they didn't really consult like the designers or the PMs or the back-end people, and then they have to like throw away the work after they've built it because it they weren't aligned up front.
00:07:28.160 --> 00:07:32.079
So this metaphor of like you can't nine women can't make a baby in one month.
00:07:32.240 --> 00:07:33.920
Like it's not possible.
00:07:34.079 --> 00:07:34.240
No.
00:07:36.480 --> 00:07:38.000
Unfortunately, unfortunately, I know.
00:07:38.079 --> 00:07:39.199
I would put labor.
00:07:39.680 --> 00:07:46.399
Um, but you know, it in order to do big things, big ambitious things, you all need to be aligned together as one.
00:07:46.480 --> 00:07:51.199
You can't split up the work amongst all of you and think that it's going to lead to some cohesive product in the end.
00:07:51.680 --> 00:07:52.160
Awesome.
00:07:52.399 --> 00:07:58.160
Uh one of the other things I think most people will probably agree that humans uh are similar to this as well.
00:07:58.319 --> 00:08:01.759
But you mentioned that agents are shit at CSS.
00:08:02.319 --> 00:08:07.680
Um and I and I would say that's because they've learned very, very well from us being really bad at CSS as well.
00:08:07.920 --> 00:08:15.680
Uh but honestly, what what would you say the current generation of models are typically very bad at that we're not acknowledging as a as an industry?
00:08:16.000 --> 00:08:22.160
Yeah, I feel like I I do hear people complaining about agent design skills as I think the term they would roll it under.
00:08:22.319 --> 00:08:27.600
Um and when I hear people say they're bad at design, what I'm actually hearing is models are trained on language, right?
00:08:27.680 --> 00:08:29.680
And and of course on code, so they're trained on CSS.
00:08:30.079 --> 00:08:36.320
But the way that a human designer like looks at designing something, they are mostly working with visual input.
00:08:36.399 --> 00:08:39.200
They're working with what we would think of as like classical graphic design.
00:08:39.360 --> 00:08:43.919
So you are trying to say, okay, what's important for someone to see on this page and what do they need to do?
00:08:44.080 --> 00:08:50.960
And then in order to achieve that, you use things like space, contrast, priority, color, grouping.
00:08:51.200 --> 00:09:02.960
Um, you like use the principles of how humans interpret objects and colours and values in space to like make the person see the right thing in the right order, click the right button, understand what's on the screen.
00:09:03.120 --> 00:09:05.919
Um and agents haven't been trained on visual inputs.
00:09:06.000 --> 00:09:16.159
Like we can do, you know, image generation and pass images to them, but they're really not good at translating like visual reasoning into code in the way that a traditional human designer does, right?
00:09:16.399 --> 00:09:21.440
Um, so sure, it doesn't help that the CSS is all probably pretty pretty crap in the training data.
00:09:21.519 --> 00:09:30.240
Um, but I think it's much more this missing piece of that they're not multimodal enough, they're too heavy on language, and so they're unable to like see what we can see.
00:09:30.320 --> 00:09:41.279
So I often, when working with models, trying to prompt them to do designs, they will just like make elements overlap because they can't see, even when they take screenshots, they don't
really understand how much space needs to be between things.
00:09:41.360 --> 00:09:43.919
Like, what does it mean for something to be visually aesthetic?
00:09:44.000 --> 00:09:52.159
They don't have an understanding of aesthetics, especially because it's um historical and cultural in the sense that like what we think looks good changes over time.
00:09:52.639 --> 00:10:02.720
So there was an aesthetic in like 2010 that if you had a website in that style now, we would think of as very old school, like, oh, that looks like you sliced that sidebar out of Photoshop and like stuck it in, you know, like we used to do.
00:10:02.960 --> 00:10:14.559
Um and now the aesthetic is like linear verselle, clean, kind of modernist, but that will change over time, so we can't even like bake into their weights like what good aesthetics are because that evolves.
00:10:14.639 --> 00:10:14.799
Yeah.
00:10:15.039 --> 00:10:27.039
But we might be able to do it through, you know, skills or just like giving them more relevant data for models at a certain time period, but it's like a a more fundamental issue with the fact they're so language-based and they don't have this visual reasoning capacity.
00:10:27.279 --> 00:10:27.440
Yeah.
00:10:27.679 --> 00:10:35.200
Do you think do you think the visual reasoning will need to uh uh account for or cater for agents as well these days?
00:10:35.360 --> 00:10:41.759
Because more and more agents tend to almost be going to the browser to do things at times where APIs and things like that don't exist.
00:10:41.919 --> 00:10:44.000
Um is that is that something we need to cater for?
00:10:44.080 --> 00:10:49.120
And I guess a follow-up question is in terms of what you see as fixing this, yeah.
00:10:49.360 --> 00:10:57.919
Do you think general models can will learn to do this, or do we need like some kind of specific model to uh I think we'll probably get general models there.
00:10:58.000 --> 00:11:00.799
I assume that the labs are doing more multimodal training.
00:11:00.960 --> 00:11:04.320
Like I assume, you know, I mean, given the speed of things, maybe two years.
00:11:04.480 --> 00:11:05.039
Yeah, yeah.
00:11:05.600 --> 00:11:06.720
If it's slow, maybe five.
00:11:07.120 --> 00:11:11.360
I definitely am not in the camp of like, oh, models will never overtake designer jobs.
00:11:11.600 --> 00:11:17.840
I I don't know that they'll take their jobs, but like in the same way that we've been able to automate away large chunks of the software engineering process.
00:11:18.000 --> 00:11:22.240
Well, I should say like specific tasks within that uh job in the same way.
00:11:22.399 --> 00:11:40.480
I hope actually there's a future where I can describe what I'm trying to achieve to an agent and the interface it makes is at least usable, like baseline aesthetic, there's enough space between things, the text is readable, there's not a purple gradient on it, like there's like some baseline that I think we're really near and within a year
or two we'll get to.
00:11:40.879 --> 00:11:46.720
Um and then the question about like, do you do you think like agents will need to browse visually?
00:11:46.960 --> 00:11:57.120
I feel like we're mostly just gonna redesign the webinar websites, and I know there's other ways they could browse, but I think we'll just they'll have APIs and they'll just like use agent methods because text will always be faster.
00:11:57.440 --> 00:12:10.720
Visuals are faster for humans because our you know brains are built differently to, you know, I say agent brains, not the same, but the way we process information is different to them, and we like visuals and space and gesture and movement in real time, and they're very like static language, you know?
00:12:11.360 --> 00:12:14.799
Yeah, just give me one line with no spaces, no white space, yeah, yeah.
00:12:15.279 --> 00:12:22.639
Um okay, one other thing that you mentioned is boundary objects, and and and you borrow this from Susan Lee uh Starr's sociology.
00:12:22.720 --> 00:12:25.279
Um so most engineers have never heard that term.
00:12:25.440 --> 00:12:26.080
Yeah, I'm sure they haven't.
00:12:26.320 --> 00:12:31.519
Tell us a little bit about what it means and and why um I guess agent MD files fail that test.
00:12:31.840 --> 00:12:32.000
Yeah.
00:12:32.159 --> 00:12:34.480
So yeah, so this is like a classic sociology paper.
00:12:34.559 --> 00:12:35.759
Like I studied anthropology.
00:12:35.840 --> 00:12:37.600
This is like one of the things you read in 101.
00:12:37.759 --> 00:12:47.200
Um, and it's from 1989, it's not that old really, but it was the study of how um Berkeley's Museum of Vertebrate Zoology first got started, which was like early 1900s.
00:12:47.360 --> 00:12:53.360
And these sociologists studied how many disparate groups of people came together to found this museum.
00:12:53.440 --> 00:13:02.000
So you've got academics, you've got funders, you've got biologists, you've got like trappers and naturalists, and they all had like different levels of uh training and expertise.
00:13:02.080 --> 00:13:02.960
They had different priorities.
00:13:03.120 --> 00:13:09.279
Some people care, of course, more about funding and and rules, and others are just concerned about like biology and capturing specimens.
00:13:09.840 --> 00:13:14.240
But they had effective boundary objects to collaborate, is the theory that they worked with here.
00:13:14.320 --> 00:13:18.399
It's like they had specimens and labels and field maps, and everyone read these objects differently.
00:13:18.480 --> 00:13:22.399
They like worked differently for each group, but they enabled effective collaboration.
00:13:22.480 --> 00:13:24.559
And you could point tons of examples of this out in the world.
00:13:24.720 --> 00:13:34.159
You just think about any huge project that brings together many different kinds of cultures and communities of people, they need effective objects to collaborate around.
00:13:34.480 --> 00:13:40.080
Um, but
the objects can mean different things to different people and take different forms for each kind of group that it interacts with.
00:13:40.320 --> 00:13:42.559
And this applies to to humans and agents pretty well.
00:13:42.639 --> 00:13:53.840
If we treat agents as a kind of being with a kind of way of seeing the world, and it's a very different to the way humans see the world and think, and we don't need to perfectly understand each other's worlds because that's probably impossible.
00:13:54.159 --> 00:14:02.879
Um I always kind of refer to the fact that you know um we are humans embodied and we like talk in real time in space, we're very social, agents have none of these qualities.
00:14:02.960 --> 00:14:09.039
You can give them cameras and mics, but they're still not going to understand what it is to be an embodied being in the way a human would.
00:14:09.440 --> 00:14:10.960
Um but they don't have to.
00:14:11.279 --> 00:14:24.080
As long as we have effective what what we currently use is right, yeah, agent MD files, skill files, MCP servers, effective boundary objects to collaborate with them, then we can actually achieve things together without needing perfect shared understanding.
00:14:24.480 --> 00:14:26.080
So is that context like an agent MD?
00:14:26.159 --> 00:14:27.679
Is that an effective boundary object?
00:14:28.000 --> 00:14:34.960
Uh for the moment it's sort of okay, but I don't think I think um it serves agents well because it's it's a text file.
00:14:35.200 --> 00:14:37.519
Everything that we do is like very text heavy.
00:14:37.600 --> 00:14:39.759
We are reading these huge outputs in the CLI.
00:14:39.840 --> 00:14:50.480
Even the fact that current agent coding tools are so CLI focused, I think is like preferencing the agents' preferences and not the humans in terms of understanding and being able to communicate.
00:14:50.799 --> 00:14:55.120
Um so we are having to read and write masses of text, which is not very effective for humans.
00:14:55.279 --> 00:14:57.840
We are much faster visual, spatial, social reasoners.
00:14:58.240 --> 00:15:04.399
It would be much more effective for you and I to like plan out some complex architecture problem with a whiteboard in real time, right?
00:15:04.559 --> 00:15:08.240
Drawing diagrams, talking, pointing to things, moving around.
00:15:08.399 --> 00:15:15.840
It's much less effective for us to like type words into a document and that to be our way of expressing what we want and problem solving together.
00:15:16.320 --> 00:15:28.960
Um so the the big thing I'm trying to push for is like we like there are fine boundary objects, like these text files, but we could make them preference humans far more by making them social, real-time, interactive, visual.
00:15:29.039 --> 00:15:32.639
Um, and I when I say interactive, I mean like it's not a static text file.
00:15:32.799 --> 00:15:38.960
Like we have beautiful like systems like the web and browsers where we could make everything interactive.
00:15:39.039 --> 00:15:48.080
You could be able to like po poke around an interactive diagram, make a a mock-up of your UI and compare it to what you might want and move sliders and move little things.
00:15:48.320 --> 00:15:54.399
I like in design we make jigs a lot, which are like little little control panels just to solve one particular problem.
00:15:54.879 --> 00:16:02.879
These are the sorts of things when I say visual interactive that I mean is like make this stuff easier for humans to understand in a way that a wall of text is not.
00:16:02.960 --> 00:16:06.320
It's very difficult for us to comprehend complex systems in walls of text.
00:16:07.600 --> 00:16:18.799
And this kind of like leads on to the final quote, which you'll be happy to hear, uh, which is um when you described humans and agents as looking at each looking at each other through a keyhole.
00:16:19.039 --> 00:16:20.240
Uh and this is really interesting.
00:16:20.320 --> 00:16:30.799
It's kind of like that, you know, that problem of uh of really not being able to kind of like see as much or understand as much of the the the entire entity as uh from both sides as as as you might want.
00:16:30.960 --> 00:16:33.440
Who whose job is it to, I guess, widen that keyhole?
00:16:33.519 --> 00:16:34.879
Is it the the model builders?
00:16:35.039 --> 00:16:37.200
Is it the tool builders like yourselves?
00:16:37.440 --> 00:16:44.480
Um yeah, who who who or is it the humans, I guess, learning better how to prompt, how to provide that information and context to the agents?
00:16:44.879 --> 00:16:47.440
I think it's it's mostly tool build builders, I think.
00:16:47.519 --> 00:17:00.720
I would maybe include model builders in this in the sense of like actually the moment the model builders and the tool builders have so much overlap, and you if you think about the fact that Claude and and um chat GPT really dominate in terms of like they kind of own the interface and the model.
00:17:00.960 --> 00:17:07.119
Um not to say that there aren't other players in the space, like of course GitHub Copilot is another one that like gives you tools but doesn't own the models.
00:17:07.599 --> 00:17:17.920
Um at least in terms of widening the keyhole that letting letting the agent see more of our world, uh at the moment it is like we're just like funneling text through this small hole.
00:17:18.319 --> 00:17:25.599
Um I mean I can kind of touch on like interpretability is like the study of how we understand how models think, and that's much more what the model companies do, right?
00:17:25.680 --> 00:17:27.119
Because from the outside we can't do that.
00:17:27.200 --> 00:17:32.160
That is like foundation labs are really the only ones in a position to do meaningful interpretability work.
00:17:32.480 --> 00:17:36.160
And I think that stuff is important to s for us to see how agents reason.
00:17:36.480 --> 00:18:02.240
But to get back to the boundary objects thing, it's actually not as
important for us to understand their worlds more comprehensively, or even them ours, as much as it's important for tool builders to make these kind of environments that humans can do good thinking in and then create artifacts that you can pass off to agents that are optimized for the agents as well, because they want to, you know, they can happily r deal with reams of text, just tons and tons of files, like nested skills that just contain all kinds of things.
00:18:02.480 --> 00:18:04.559
But that is really difficult for humans to work with.
00:18:04.720 --> 00:18:18.160
I think it's the responsibility of the tool builders to make better interfaces and environments for humans to like figure out what does the system does, what do I want it to do, like do how do I design a reliable architecture that will like stand the test of time?
00:18:18.559 --> 00:18:21.519
Those need different interfaces to just like text editors.
00:18:21.839 --> 00:18:27.599
Yeah, it's down to the developer experience or the human experience now in terms of how we actually how we actually interact.
00:18:27.920 --> 00:18:29.759
Um let's talk about planning.
00:18:30.000 --> 00:18:31.759
Let's talk about planning and design, I guess.
00:18:31.920 --> 00:18:42.160
Um how does that change when agents actually, rather than um I guess being on the other side of the keyhole where we're just providing information to them, how does that change when they actually become collaborators?
00:18:42.480 --> 00:18:48.720
So um roles, responsibilities, how they help and how they become part of the team.
00:18:48.880 --> 00:18:56.079
Is that something that we want, or is that something that you know we actually want it to be much more human-oriented and actually maybe we get a bit of input from agents?
00:18:56.319 --> 00:18:58.559
What's the best dynamic you think we should have?
00:18:58.960 --> 00:19:02.400
I definitely think agents are active participants in planning is really good.
00:19:02.640 --> 00:19:06.880
Because it kind of gets into not like you know, I think we should have a thousand layers of agents and everything.
00:19:07.200 --> 00:19:19.519
There's like the agent over here who might implement something, but they can be far away from the fact that like when you are trying to plan something out, there's a whole stage where you need to like understand the current state of the code base um and then figure out what you want to change.
00:19:19.599 --> 00:19:25.279
And there can be all kinds of helper agents in there, and we already do this, I think, in a in a way that is inefficient.
00:19:25.519 --> 00:19:30.160
Where like we're in our CLI, we're like, okay, I want to plan something, like go into plan mode or whatever.
00:19:30.319 --> 00:19:33.359
And the agent's like, okay, I'll spend more time like asking you questions.
00:19:33.519 --> 00:19:42.240
Like, I'm sure if you've you've probably used like Matt Pecock's grill me skill, like very infamous at this point, where it asks you like 40 questions in a row, all multiple choice in this text editor.
00:19:42.400 --> 00:19:51.680
And for almost every question, I feel like each could be its own like research thread or like rabbit hole I need to go down to say, like, okay, you're asking me to pick between a bunch of NPM libraries.
00:19:51.759 --> 00:20:02.079
I don't know, give me like a comprehensive report of how these compare and how they relate to the problem I'm trying to solve, or like you're asking me about visual design, show me visual design options, like show me my design system.
00:20:02.240 --> 00:20:04.720
Do we need to change the design system for this design to work?
00:20:04.880 --> 00:20:11.440
Like, every one of those is a complex problem to dig into, and you can't solve it with a multiple choice text question.
00:20:11.839 --> 00:20:14.160
You need to make it its own rich artifact.
00:20:14.480 --> 00:20:15.839
But this is where agents can help.
00:20:15.920 --> 00:20:22.000
Like every one of those could have a bespoke interface designed for you with agents, which is also why I care about agents getting better at visual design.
00:20:22.079 --> 00:20:28.640
It's like I really do want generative UI to work out, not for the end product, but for the interim developer experience.
00:20:29.440 --> 00:20:36.240
Um, so like I just feel like this like very textual, simple interface isn't cutting it.
00:20:36.480 --> 00:20:39.279
And I want agents in there proactively doing stuff too.
00:20:39.440 --> 00:20:42.799
Don't wait for me to prompt you, be like, oh, I see where you're going.
00:20:42.880 --> 00:20:44.319
You're trying to work on something with auth.
00:20:44.400 --> 00:20:49.599
I'm gonna go research a bunch of like cutting-edge practices or like what are the new libraries you could be using.
00:20:49.759 --> 00:20:55.519
I'm gonna go do web search ahead of time so that by the time you've reached this topic, I know we're gonna get to in the plan.
00:20:55.680 --> 00:20:59.279
I already have reports ready for you with bespoke interfaces ready to go.
00:20:59.599 --> 00:21:12.559
So a lot of I'm exploring this with Chopin is like how do we get agents to look ahead, be proactive, do the research ahead of time, present you with decisions like as you move through without you having to always be the one to like prompt it.
00:21:12.799 --> 00:21:26.319
But I think people have been not controversial about this, like you know, get the agents to prompt you because it does think like, well, the agent sometimes forgets to ask you things or like it can't it's not as good at critical thinking as we are at times unless it's prompted to be.
00:21:26.720 --> 00:21:29.920
Um so you don't want to like let the agent drive too much, I think is the worry.
00:21:30.079 --> 00:21:37.680
And at the same time, I do think agents prompting us more would actually help relieve our cognitive load, which a lot of us I think are struggling with in the age of agents.
00:21:37.839
--> 00:21:42.160
You're just constantly juggling seven different sessions and trying to remember what you've been doing.
00:21:42.400 --> 00:21:47.200
I really want agents to be like, hey, every morning, here's a summary of what we've worked on.
00:21:47.599 --> 00:21:51.279
Because of these, here I think are five good next steps you should consider.
00:21:51.440 --> 00:21:54.880
You know, it doesn't I don't have to run with them, but I want them to prompt me more.
00:21:54.960 --> 00:21:59.440
I want them to be doing more sense making across the work I'm doing over the course of a day or week or month.
00:22:00.079 --> 00:22:02.960
And saying, you're n you're forgetting this, you haven't looked into this.
00:22:03.200 --> 00:22:04.640
Good to think I looked into it for you.
00:22:04.799 --> 00:22:05.680
Here's my results.
00:22:05.759 --> 00:22:06.799
Like, I'm looking for that.
00:22:07.279 --> 00:22:09.599
And there's a couple of errors I want to dig a little bit deeper in.
00:22:10.079 --> 00:22:13.440
One, which we'll get to, is the single player versus multiplayer on that planning.
00:22:13.519 --> 00:22:17.200
Because I think the dynamic changes and how you actually interact with the agent changes.
00:22:17.359 --> 00:22:19.599
But I want to touch on junior developers.
00:22:19.920 --> 00:22:38.400
Because this is I uh what I what I think this can lead to is actually a really good path for junior developers to actually rather than you know the worry is oh a junior developer hasn't been through the trenches like like a senior developer has, they don't know the pitfalls, they don't know what to look out for or or how to design or plan well.
00:22:38.559 --> 00:23:02.319
Now, if you have an agent here helping with that planning and understanding the pitfalls, or at least having the skills or knowledge to recognise what a senior developer would do or the pitfalls that a junior developer might walk into, how much how much will will that do you think um kind of bridge the gap between the senior developer, junior developer uh wisdom that that sometimes lacks?
00:23:02.720 --> 00:23:03.359
Hopefully a bit.
00:23:03.440 --> 00:23:06.640
I mean, if we think about like pair programming must be totally dead by now.
00:23:07.279 --> 00:23:08.160
Which is how you used to learn.
00:23:08.559 --> 00:23:16.480
It's probably still alive, but it's a pairing with an agent versus thing of like everyone like vibe coding their own apps, right?
00:23:16.559 --> 00:23:22.559
They don't even know what questions to ask, they don't even know to ask about security, or they don't even know to ask about how deploying something works.
00:23:22.880 --> 00:23:23.440
Yes, yes.
00:23:23.680 --> 00:23:30.960
Um but but if you ask the agent the right questions, you can of course get good quality answers that like teach you all about those very important topics.
00:23:31.440 --> 00:23:39.200
But I would hope like in when we get agents to move into these more real-time collaborative spaces when we're doing our development work and out of these like local silos.
00:23:39.519 --> 00:23:46.079
I the hope is that juniors and like everyone on the team just gets more visibility into everyone else's agent sessions.
00:23:46.160 --> 00:23:47.359
That's like one piece of it.
00:23:47.519 --> 00:23:51.200
Is the way we've designed most of these systems is open by default within a team.
00:23:51.279 --> 00:23:53.599
So like everyone can see everyone else's agent chats.
00:23:54.079 --> 00:23:55.680
This is like we got, of course, feedback.
00:23:55.759 --> 00:24:02.960
This is a bit controversial that everyone's like, I don't want people to see what I'm asking the agents, but if you're on a team, you're building the same product, you really sh it should be open.
00:24:03.200 --> 00:24:04.640
And it should be sharing as well, right?
00:24:04.720 --> 00:24:15.599
Because you you want that level of uh you know one person not being super effective and other people being you know ineffective because they they they they don't understand the prompting just uh as much and so forth.
00:24:15.920 --> 00:24:16.559
Exactly, exactly.
00:24:16.720 --> 00:24:19.680
Seeing how other people prompt the agent, seeing what kind of questions they ask.
00:24:20.000 --> 00:24:24.640
But even if you were able to like like I'm I don't do backend or I I if I do backend, it's very bad.
00:24:24.799 --> 00:24:28.160
But I get to work with people who are like very senior backend people.
00:24:28.559 --> 00:24:50.319
If I could actually see them writing plans in real time and seeing how what they correct, what questions they ask the agent, where they ask the agent to go do research, just like I think that's the way that you would get junior developers to actually learn is to be in that sort of environment of like they see the quality of the level of plans that a senior engineer would hand off to an agent and they understand more about what's gone into it.
00:24:50.480 --> 00:24:55.119
Versus like I don't even know what they're doing at the moment, other than sitting alone with their agents not asking the right questions.
00:24:55.359 --> 00:24:55.440
Yeah.
00:24:55.680 --> 00:25:03.200
And and I wonder, I I question even if if the junior dev, let's say, even needs to know some of those things.
00:25:04.000 --> 00:25:22.640
Uh as in, needs to know to ask the agent some of those things, or whether it's actually kind of like built into the agent that other, you know, the more senior developers are asking those questions or are uh you know offering um opinions in these areas, I should make sure this is also considered here.
00:25:22.720 --> 00:25:22.880
Yeah.
00:25:23.039 --> 00:25:31.440
And I should probably tell the junior developer, designer, planner that this is what I'm doing, and this is maybe I you know this is an important area or something like that.
00:25:31.680 --> 00:25:32.720
Yeah, I'd be surprised.
00:25:32.799 --> 00:25:34.160
I'm sure
people have skills for this already.
00:25:34.640 --> 00:25:34.720
Right.
00:25:34.799 --> 00:25:36.880
It has to be it's gonna be more a guidance or skills thing.
00:25:37.279 --> 00:25:45.200
Yeah, or maybe they'll bake it into the models where like models now, I think that we're finally getting a handle on memory where it really has an understanding of your profile and what you're good at and bad at.
00:25:45.279 --> 00:25:52.960
And I think if it could recognize someone was a junior, I would hope the model was to some degree trained to m ask more proactive questions or like understand.
00:25:53.200 --> 00:25:57.119
But it conflicts with the with the rule, like you know, the human's always right.
00:25:57.440 --> 00:25:57.599
Yeah.
00:25:57.839 --> 00:26:03.359
So I don't know, do we like put junior developers in a position where they don't have as much authority over their agent?
00:26:03.440 --> 00:26:04.319
That seems troublesome.
00:26:04.799 --> 00:26:09.519
So maybe it was instead of saying you're absolutely right, maybe it would just say, Ah, you're you're probably right.
00:26:09.599 --> 00:26:10.079
Yeah, yeah.
00:26:10.400 --> 00:26:21.519
I mean, this is like uh because for a while I was a bit worried about the whole thing of of agents always validating our ideas, and like I always I wanted an agent that would like enable my own critical thinking and my own reflection.
00:26:21.759 --> 00:26:25.680
So at some point I put in my Claude system prompt, like, you should question me on everything.
00:26:25.839 --> 00:26:28.880
Like, don't let me just assume I'm right.
00:26:29.119 --> 00:26:41.920
But um, it was so extremely annoying the way that it behaved afterwards was every single thing I said, even if I was like, I'm researching new kitchen scissors to buy, it would be like, Well, you probably aren't correct about your assumptions of what you need.
00:26:42.720 --> 00:26:45.359
Just I this turn off, never mind.
00:26:45.519 --> 00:26:46.480
I regret this.
00:26:46.799 --> 00:26:47.759
I shouldn't have asked.
00:26:49.119 --> 00:26:52.240
Um let's go into single player versus multiplayer here as well.
00:26:52.400 --> 00:27:03.920
Um so as an individual, if I was to think about um planning, yeah, I you know, just interacting directly with an agent makes a ton of sense that I can kind of like ask some questions, have a good back and forth.
00:27:04.079 --> 00:27:12.240
But of course, let's say you and I were to do some planning, exactly what you said, we'll be on a whiteboard and we'll be talking and drawing and and and visually planning.
00:27:12.480 --> 00:27:15.039
How does an how does an agent best interact there?
00:27:15.200 --> 00:27:18.480
Is it is it something that they should validate in post?
00:27:18.720 --> 00:27:26.880
Is it something that they should almost like you know hear our conversation, understand our assertions and and why we're doing things, and then give commentary?
00:27:27.519 --> 00:27:28.480
What's the best
place to use?
00:27:28.799 --> 00:27:35.039
Uh okay, the thing I'm playing with at the moment is is a little bit of them like watching quietly the chat.
00:27:35.200 --> 00:27:44.319
So uh in the prototype we have in Chopin, we have like a chat and a and a document, and the document is more like the the artifact you're making, the plan you're making, and the chat is of course free form chat.
00:27:44.480 --> 00:27:49.279
But when we're when I'm planning with my engineer, like we are also usually on a call talking.
00:27:49.599 --> 00:27:54.640
And at the moment we don't have it hooked up where the agent like reads our transcript, but I do think that would be a good direction to go in.
00:27:54.960 --> 00:28:00.240
But at the moment, what I have is like we're typing in the chat, and of course, like you know, Jev is like the the hottest thing ever.
00:28:00.640 --> 00:28:04.640
Of course, the minute Jev came out, I was like, perfect, we've been doing this badly with models, let's go.
00:28:05.039 --> 00:28:09.440
Everything you type in the chat gets classified as like, is this a question, a proposal, an opinion?
00:28:09.599 --> 00:28:11.920
Uh is this a constraint on an existing decision?
00:28:12.160 --> 00:28:19.119
So the agent is constantly trying to understand what's happening in the chat, and it's putting things into the plan in structured formats.
00:28:19.200 --> 00:28:21.279
It's like marking like this is a decision you need to make.
00:28:21.359 --> 00:28:23.599
Here are the options you've laid out for that decision.
00:28:23.839 --> 00:28:26.240
Uh, here's like the goal that you've stated in the chat.
00:28:26.319 --> 00:28:35.759
So you're trying to turn this like very continuous free-form human conversation into a more structured artifact that you will work towards handing off to an implementation agent.
00:28:36.000 --> 00:28:39.039
Um, so this starts to head in the right direction for me.
00:28:39.200 --> 00:28:43.599
Like, um, we're trying to figure out how to put in like TL draw whiteboards to be able to draw us.
00:28:44.000 --> 00:28:44.160
Yeah.
00:28:44.319 --> 00:28:48.640
And I need to figure out a way to hook up something like a real-time cool transcript to it.
00:28:48.799 --> 00:29:07.279
But my ideal like planning environment is free-form chat with someone where we just like hash out the details, but the agents are these background agents classifying, and then when they spot something like we have an open question that could be a web search they need to go do, I want them to proactively write that research report, put it in line in the plan.
00:29:07.440 --> 00:29:09.039
So that's the next piece we kind of want to work on.
00:29:09.279 --> 00:29:10.079
Yeah, really interesting.
00:29:10.240 --> 00:29:15.920
And I think the handoffs that you mentioned there, or when the agent interacts, is super, super critical.
00:29:16.079 --> 00:29:24.480
Um, as a parallel, as an aside, I've been I've been using Claude, I must admit, on voice uh on voice chat while I've been cooking recently.
00:29:24.720 --> 00:29:31.680
And it's been, it's, it's, it's not great at timings, uh, but it's good with it's good with telling me, oh yeah, you should add this or that.
00:29:31.839 --> 00:29:35.599
I kind of like show at the fridge and I'm like, oh yeah, you can add this and this and this and add that.
00:29:35.920 --> 00:29:46.559
And one of the it's great when it's just me, but then when my kids walk in or my wife walks in and just say something, all of a sudden Claude's like answering back in my ear, uh, and it's like, okay, that's not great.
00:29:46.640 --> 00:29:48.640
I don't want you to interact at this point.
00:29:48.799 --> 00:30:02.960
And I guess the one of the one of the questions here would probably be like recognizing there is a there is a point in time when I want an answer from you or I want a contribution from you, and it's almost like keep like milestones where it's like, okay, we've we've got to this stage.
00:30:03.039 --> 00:30:03.680
What have we forgotten?
00:30:03.839 --> 00:30:04.400
What do we need to add?
00:30:04.480 --> 00:30:06.960
What are we what are what are the constraints should we be aware of?
00:30:07.039 --> 00:30:09.839
Yeah, and then get it to get it to come in at that point.
00:30:10.079 --> 00:30:10.880
Yeah, yeah, I know.
00:30:10.960 --> 00:30:13.680
I this is why I've struggled with that real-time chat interface.
00:30:13.759 --> 00:30:16.000
Like I've tried it with chat GPT, but it's the same thing.
00:30:16.079 --> 00:30:23.359
It's like someone I'll be like out in the park walking with my kid and then trying to talk to it, but then of course, you like look weird like you're chatting to this age.
00:30:24.400 --> 00:30:28.880
Someone gets too close or they come talk to you, you're like awkwardly like hang up a call with an age.
00:30:29.039 --> 00:30:31.039
I don't know, the whole thing socially doesn't work for me.
00:30:31.119 --> 00:30:32.880
Um, and it's this really I love it.
00:30:33.119 --> 00:30:33.599
I love that.
00:30:33.759 --> 00:30:35.119
I really love it socially, yeah.
00:30:35.279 --> 00:30:44.240
Yeah, um, but it is like I think the way I've always tried to design these like these collaborative like agents in real-time systems is agents as in the background.
00:30:44.400 --> 00:30:45.680
I don't want them interrupting, yeah.
00:30:45.839 --> 00:30:47.440
Especially if I'm talking to a coworker, yeah.
00:30:47.599 --> 00:30:49.920
Like they do not have the right to like come in.
00:30:50.079 --> 00:30:56.960
Do you know, like in the beginning when we had all these um agent transcribers in like Zoom and stuff, and they would like look like a person in the screen.
00:30:57.279 --> 00:31:02.559
And I was always like, they do not have the prominence to be a person in this call, they are like a background helper.
00:31:02.799 --> 00:31:07.440
Um, so I haven't really explored much where the agent interrupts in any sense.
00:31:07.599 --> 00:31:13.519
I definitely want it to be like it's doing continuous background work in a way that you can always uh discard it.
00:31:13.759 --> 00:31:17.440
Because it might do some research that you're like actually not relevant.
00:31:17.519 --> 00:31:22.400
Like my opinion, I already have experience in this, I'm picking this framework, I don't want to hear about your research report.
00:31:22.640 --> 00:31:27.519
I think it's something like we have to assume all the agent work can be like just disposable.
00:31:27.680 --> 00:31:36.000
Like it'll become cheap enough and intelligence will be cheap enough that this is all just like fine for it to constantly churn and try to suggest things and then we can dismiss as needed.
00:31:36.319 --> 00:31:43.359
Um but I don't I don't I haven't yet figured out how they could really be active participants in a way that doesn't feel invasive, annoying.
00:31:44.160 --> 00:31:50.160
Yeah you almost need like that JEV decision that the agent thinks, should I interrupt it or should I not interrupt it?
00:31:50.319 --> 00:31:56.720
Am I can I can I contribute something that the that might that the users here would actually find valuable at this at this point?
00:31:56.960 --> 00:32:04.880
Yeah um let's talk a little bit about handoffs, huge like human to agent handoffs and agent to multiple human handoffs as well.
00:32:05.039 --> 00:32:13.599
Um I guess where where do you find what do what do you find is the biggest problems with handoffs today, maybe around content speci uh context specifically.
00:32:14.000 --> 00:32:14.319
Yeah.
00:32:14.480 --> 00:32:19.279
Um it's interesting actually, we're getting someone to come work on Chopin who's really into like formal verification.
00:32:19.680 --> 00:32:26.640
So so at the moment, right, we all like write this plan and we're like, you know, you might ask the agent, like, uh, have I thought through everything?
00:32:27.119 --> 00:32:30.240
And like it's usually just gonna be like sometimes actually it does have good answers.
00:32:30.400 --> 00:32:32.640
Like you can always try and sense check yourself.
00:32:32.960 --> 00:32:34.880
But it's it's almost certain, right?
00:32:34.960 --> 00:32:38.640
You hand this plan off, and then the agent always finds things during implementation, right?
00:32:38.799 --> 00:32:44.799
I always tell it, like, okay, you can make decisions as you go because you're going to need to, um, if I'm doing it where I'm not around.
00:32:44.880 --> 00:32:45.920
I just want you to log them.
00:32:46.000 --> 00:32:52.960
At the end, I want like a really nice visual report where you summarize everything you've done and why you did it, and a list of all the decisions you made along the way.
00:32:53.119 --> 00:32:57.119
Um, because we just it's so hard to comprehensively cover everything up front.
00:32:57.200 --> 00:33:02.400
It always finds bugs or edge cases, or like, oh, we didn't consider what happens in this in this state.
00:33:02.640 --> 00:33:17.279
Um, formal verification systems are supposed to help with this, where like you might take a uh a plan that you've made with a coworker and it's to your the best of your ability comprehensive, but then try and turn it into a formal spec where it really tries to cover every single possible use case.
00:33:17.359 --> 00:33:29.279
And that also means that after the implementation, you can compare it to the formal spec and like validate whether it actually did the work it was supposed to in a way that's harder to do in kind of more free-form like natural language plans.
00:33:29.519 --> 00:33:35.359
So that's one interesting direction is like could you just like computationally verify that the plan was completed to a certain degree?
00:33:35.680 --> 00:33:44.400
And the uh yeah, I do think the other one is just like their reporting on the other side um helps with you figuring out like okay, this is this is what went wrong along the way.
00:33:44.960 --> 00:33:45.200
Yeah.
00:33:45.519 --> 00:33:50.720
And and how much does this uh kind of lean into shared understanding before delegation?
00:33:50.799 --> 00:33:58.240
Because I guess I guess you know, is that shared understanding should that be between multiple agents, many humans?
00:33:59.119 --> 00:34:01.119
Where do you draw that line between shared understanding?
00:34:01.359 --> 00:34:17.920
Yeah, um, I think we've explored this a little bit, but probably not enough in the sense that I know in some companies, like you really want everyone to not have like seen or checked the plan, but to have consensus that before you hand it off, especially um, I think if you're being ambitious about your agentic runs, they can be really expensive.
00:34:18.239 --> 00:34:25.199
If you're really like, okay, we've planned this feature thoroughly, like we want to run, like I mean, I love orchestrator agents, I know everyone does, right?
00:34:25.280 --> 00:34:33.199
But I hand always hand mine to a powerful orchestrator and I say you use you know less powerful models as needed to do the sub agents to implement and test and review everything.
00:34:33.360 --> 00:34:36.800
Um, so you have your orchestrator who you know is gonna run the show.
00:34:36.960 --> 00:34:40.559
Uh, you want to make sure that you haven't planned incorrectly.
00:34:40.639 --> 00:34:49.760
You want to make sure the plan isn't like um when the when the run's complete and you've already spent the tokens, you don't want the designer to come back and be like, well, that doesn't fit into our system.
00:34:50.079 --> 00:34:54.159
Or you don't want your infra person to come in and just be like, we can't deploy that for some reason.
00:34:54.320 --> 00:35:01.519
Um, getting your whole team to sign off on a plan, I think, is more of a heavyweight thing, but I imagine you would do in bigger companies.
00:35:01.679 --> 00:35:15.119
If you're not a small startup and you're spending significant numbers of tokens on like really
good quality implementation and checks and stuff, you want everyone to be like, we agree this is the right thing to build, it's the right time to build it, PMs agree like this isn't gonna flop in the market.
00:35:15.280 --> 00:35:23.760
Like I I think we'll get see more of this like formal approval up front before we hand everything off to implementation agents.
00:35:24.000 --> 00:35:27.280
Yeah, and and so that's very much human kind of like agreement.
00:35:27.360 --> 00:35:27.519
Yeah.
00:35:27.679 --> 00:35:35.280
Do you do you see that as almost like almost spec driven, or do you see this as something that's kind of like a combination of other other pieces?
00:35:35.360 --> 00:35:42.239
Because I know you know Yeah, uh the team, the GitHub Next team has has done some previous research before I I joined on spec driven development, yeah.
00:35:42.400 --> 00:35:44.960
Where they were like the spec's the software, like how is it different?
00:35:45.440 --> 00:35:48.320
I think that can become more true with the formal verification methods.
00:35:48.480 --> 00:35:57.920
I think with natural language I'm a bit more skeptical because it's like what is code except for precisely stated, you know, like ha rules of how software works.
00:35:58.159 --> 00:36:02.159
And the spec is like one level up from that, but people will argue like that's what code is.
00:36:02.239 --> 00:36:03.440
It's the code is the spec.
00:36:03.679 --> 00:36:05.760
Um, which I I kind of agree with.
00:36:05.920 --> 00:36:10.559
Um but yeah, I do I do feel like we're just inevitably moving towards towards spec driven development.
00:36:10.639 --> 00:36:12.719
Like, how much is anyone really gonna read the code?
00:36:12.880 --> 00:36:16.719
Again, as long as like the tests verify that it does what it says it does.
00:36:17.039 --> 00:36:21.840
I yeah, I I feel like we just move into interacting with these higher-level artifacts.
00:36:21.920 --> 00:36:32.239
So these like plans and these like um after an agent completes a run, what we would currently do in like a PR description, but we obviously need new nouns and like new primitives there.
00:36:32.400 --> 00:36:39.360
Um, but some sort of report of after your all your orchestrator agent and all your implementation agents have done some huge run.
00:36:39.599 --> 00:36:41.920
It's like you don't want a line-by-line diff.
00:36:42.239 --> 00:36:50.800
Yeah, you want a really high-level understanding of what happened, and and you can also have agents, you know, verify that final report actually reflects what the agents did.
00:36:50.880 --> 00:36:53.360
You can have many layers of checker agents all through this.
00:36:53.519 --> 00:37:03.920
Um, but you'll just be reviewing these high-level inputs and high-level outputs, and you won't have to worry as much about all the noise down below because we'll have figured out ways to actually make this quite rigorous and testable.
00:37:04.239 --> 00:37:06.960
Hey everyone, hope you're enjoying the episode so far.
00:37:07.199 --> 00:37:15.360
Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development.
00:37:15.519 --> 00:37:20.880
Whether that's talking about the latest tools, the most efficient workflows, or defining best practices.
00:37:21.119 --> 00:37:24.719
But for whatever reason, many of you have yet to subscribe to the channel.
00:37:24.960 --> 00:37:31.679
If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favour and hit that subscribe button.
00:37:31.840 --> 00:37:39.199
It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.
00:37:39.519 --> 00:37:41.199
Alright, back to the episode.
00:37:41.440 --> 00:37:44.960
Let's talk about implementation and how we actually now build with agents.
00:37:45.199 --> 00:37:48.480
Um I I said I wouldn't quote you again.
00:37:48.639 --> 00:37:49.519
Uh I lied.
00:37:49.599 --> 00:37:51.360
I I think this is one of your quotes.
00:37:51.519 --> 00:37:56.159
Um, in a world of fast, cheap software, quality becomes the new differentiator.
00:37:56.320 --> 00:37:57.119
Was that one of yours?
00:37:57.519 --> 00:37:59.280
I feel like everyone said that at this point.
00:37:59.519 --> 00:38:01.440
I don't know if that was me, but I've definitely said it.
00:38:01.840 --> 00:38:02.320
Okay, that's cool.
00:38:02.400 --> 00:38:02.719
That's cool.
00:38:02.960 --> 00:38:03.920
We'll say it's one of yours.
00:38:04.239 --> 00:38:09.840
Um so quality is notoriously hard to be able to measure, I guess, and validate.
00:38:10.000 --> 00:38:13.840
Um or even reward in companies as well.
00:38:14.000 --> 00:38:31.280
How how do you get a team or an exec to um to pay for, I guess, the craft that they don't see in a dashboard when speed and delivery is becoming, I guess, one of the one of the more uh sought-after pieces, efficiency in in in organizations?
00:38:31.679 --> 00:38:38.239
I think it'll happen naturally because I think before you really had to fight for craft for the time to finally tune something.
00:38:38.480 --> 00:38:43.840
But this was in the time where software was was compared to what we have now, infinitely slow.
00:38:44.239 --> 00:38:49.760
Just you would spend weeks planning something and then like months implementing it, even like a dashboard or something.
00:38:50.000 --> 00:39:00.079
Which now, okay, you can bang out a version in an afternoon that sure is not gonna be as like um verified that it's like good quality, and there'll probably be lots of bugs to solve.
00:39:00.239 --> 00:39:11.360
But we're in the age where you can assign a bunch of like autonomous agents to just like run a bug fix program, like go try and use the software on a loop, find every bug, file a bug, fix it, have another agent review it.
00:39:11.440 --> 00:39:15.360
Like, this is like the software factory stuff is really solving a lot of this quality control to it.
00:39:15.599 --> 00:39:17.519
And what does debt what is debt now?
00:39:18.159 --> 00:39:20.400
Yeah, it's just tokens, right?
00:39:20.719 --> 00:39:22.320
It is what is tokens allow down.
00:39:22.880 --> 00:39:28.960
Um I feel like craft, yeah, craft becomes the differentiator, but I do actually think craft is more within our reach now.
00:39:29.360 --> 00:39:35.119
I certainly feel far more empowered as both a designer and developer to like today than even six months ago.
00:39:35.360 --> 00:39:41.519
Um, that I can do much more ambitious things and to a much higher quality level than previous me could have ever done.
00:39:41.920 --> 00:39:43.840
It's just I just was not as capable.
00:39:44.000 --> 00:39:48.719
We would all just make kind of like okay software, frankly, in the past.
00:39:48.960 --> 00:39:52.480
I don't think quality is degrading from what I see around.
00:39:52.800 --> 00:39:56.800
You could you can look at like people who like vibe up slop, of course it's bad.
00:39:57.039 --> 00:40:01.599
But really, if you look at like the big companies, it's not getting worse the software.
00:40:01.679 --> 00:40:04.400
It's actually getting bug bug reports are going down.
00:40:04.480 --> 00:40:06.159
It's like things are shipping faster.
00:40:06.400 --> 00:40:14.079
We're going through a weird transition period where where execs are demanding more speed because that you have agent tokens, you can just build stuff faster.
00:40:14.320 --> 00:40:29.360
But I think they're quickly gonna realize like deploying agent tokens effectively towards like reducing bugs and also like leveling up the quality of your interface and like being able to explore more options and like user test them with more people and then figure out what actually works.
00:40:29.519 --> 00:40:31.119
It's so cheap to prototype these days.
00:40:31.199 --> 00:40:33.280
I feel like we get faster insights into stuff.
00:40:33.360 --> 00:40:33.519
Yeah.
00:40:33.840 --> 00:40:42.480
Um I mean, I don't know how you measure it, but I hope someone can come up with some quantitative measurements for like how the quality of software improves, but I have no doubt it's happening.
00:40:42.719 --> 00:40:42.960
Yeah.
00:40:43.280 --> 00:40:57.280
So if we if we look at what's speeding up and what's slowing down and where the bottleneck is, then the uh how something is actually built becomes cheap and it's uh probably largely defined by models and agents and and and and loops.
00:40:57.599 --> 00:41:14.639
Um the the sh the the I guess the the how we build something in terms of the more of the planning and the design is is likely more human-led or or certainly human signed off and uh probably a genetic assistance needed, required, used.
00:41:14.800 --> 00:41:19.519
Uh but one level above that, the should we build it, I guess.
00:41:19.760 --> 00:41:20.079
Yeah, yeah.
00:41:20.320 --> 00:41:28.480
That that becomes the the harder thing because in a world where in a world where building becomes cheap and we can um and everything's possible, what do we do?
00:41:28.639 --> 00:41:28.800
Yeah.
00:41:28.960 --> 00:41:32.159
Who who's equipped on a team to really answer that?
00:41:32.320 --> 00:41:36.719
Is that the engineer, the architect, or is it is it more leaning into the PM?
00:41:36.960 --> 00:41:38.239
Yeah, I think it's the PM.
00:41:38.400 --> 00:41:38.800
I don't know.
00:41:38.880 --> 00:41:46.320
It's like I think like PMs get a bad rap, but like I worked with some really great PMs where like they make far better decisions than any of the engineers about what we should do.
00:41:46.960 --> 00:41:56.559
Uh and it really is when I say PM, I just mean someone who actually is like doing thorough research on the state of the market, on like what other products are available, doing comparisons.
00:41:56.639 --> 00:42:02.960
I like when it's someone's full-time job just to care about like are we building the right thing in the right way?
00:42:03.599 --> 00:42:06.800
They kind of do become the arbitrators of like what we should build.
00:42:07.039 --> 00:42:12.960
I think before you would have had designers and engineers more helping make those decisions because they could speak to feasibility more.
00:42:13.039 --> 00:42:17.280
But now that you can be more ambitious and build more stuff, I feel like there's less chance for them to say no.
00:42:17.679 --> 00:42:20.639
In the past, it would be like, you know, wouldn't it be nice if we could build this?
00:42:20.800 --> 00:42:22.880
Was like most product conversations.
00:42:23.119 --> 00:42:36.800
And now it is possible for a small team to build m much more ambitious stuff that I feel like you really can go after the ideal version of things and not always we had to make the compromise and build the crappy skateboard and not the car kind of thing.
00:42:37.119 --> 00:42:41.840
Now I feel like maybe you can't, you know, always build the car, but you can build something like the bicycle.
00:42:41.920 --> 00:42:51.440
I don't know what the metaphor is here, but you can build more ambitious things and should let PMs kind of like guide guide the decision making of what to build.
00:42:51.840 --> 00:42:55.599
And do you see how do you how do you see those two roles like scope creeping?
00:42:55.679 --> 00:42:58.559
I guess let's call it scope creep, given that we're talking about PMs.
00:42:58.719 --> 00:43:08.000
But um how how do you see the the the the PM role given you know the the development role in the engineering world is becoming more accessible?
00:43:08.239 --> 00:43:08.639
Yeah, yeah, yeah.
00:43:08.880 --> 00:43:13.840
Um I know there's this thing of like we won't need developers, the PMs will just like prompt it.
00:43:14.159 --> 00:43:17.679
Or is it or is it that the engineers actually become more of the PMs as well?
00:43:18.320 --> 00:43:19.679
Is there a merge
happening there?
00:43:20.000 --> 00:43:20.159
Maybe.
00:43:20.239 --> 00:43:21.679
I still feel like you need distinct roles.
00:43:21.760 --> 00:43:31.039
Like, unless you're gonna say the engineer's gonna become really good at user interviews and market research and product comparison and strategy, I feel like that's too much to burden one person with.
00:43:31.119 --> 00:43:37.519
We always want to try and collapse roles in this industry, but the engineer still has to be in charge of like, is this architecture scalable?
00:43:37.679 --> 00:43:41.599
Are we gonna like fall over if a hundred thousand people like join at once?
00:43:41.760 --> 00:43:48.719
Like, I think you still need someone who's in charge of owning the engineering and like keeping really high quality product.
00:43:48.880 --> 00:43:54.559
And then that is a totally different job to use a research, market research, product strategy.
00:43:54.719 --> 00:43:59.679
And I think it it's like would be b a bad idea to collapse them because it's too much for one person to manage.
00:44:00.000 --> 00:44:06.719
And they have, I suppose, different outcomes in terms of an engineer would be more focused on the quality and the and the and the architecture and the design.
00:44:07.119 --> 00:44:13.679
Whereas the PM would actually care much, much more about the why should we build this and who's this for and focusing more on the user and the outcome of the product.
00:44:14.159 --> 00:44:15.199
Yeah, and it's the same with designers.
00:44:15.360 --> 00:44:17.599
Like designers aren't dead despite everyone going on about it.
00:44:17.679 --> 00:44:20.719
Like the designer still advocates for like what's the best interface for the user?
00:44:20.800 --> 00:44:21.599
What do they need to do?
00:44:21.679 --> 00:44:23.599
Like, how do we make this look and feel the right way?
00:44:23.760 --> 00:44:27.760
And that is again, I feel like a completely different skill set to what the PM does and the engineer does.
00:44:27.920 --> 00:44:28.079
Yeah.
00:44:28.239 --> 00:44:28.559
Yeah.
00:44:28.880 --> 00:44:45.599
Okay, so so let's kind of combine a couple of these and ask a ask a little bit of a teasing question here, which is if we could start our careers again, but in today's world, um, you know, we're we're maybe just finishing university, coming into uh coming into the world of industry.
00:44:45.920 --> 00:44:50.960
What kind of role do you feel given where AI is today?
00:44:52.000 --> 00:44:58.719
You would pique your interest in terms of in terms of you know you being you wanting to be a builder, what would your role be?
00:44:59.119 --> 00:45:02.639
Oh, I mean I've always been a prototyper really.
00:45:02.719 --> 00:45:09.519
Like from from early on, um trying to like build websites like as a as a teenager.
00:45:09.679 --> 00:45:12.239
I was always just trying to do like weird scrappy ideas.
00:45:12.400 --> 00:45:15.920
And I think I would have I I would love to be like 21 right now.
00:45:17.199 --> 00:45:24.079
Um because because I would always come up with these ideas and sketch them out or like design them in Photoshop, but like I could never take them much further.
00:45:24.159 --> 00:45:24.320
Yeah.
00:45:24.559 --> 00:45:26.239
Now I would just be flying.
00:45:26.400 --> 00:45:31.599
I would frankly, I would get myself into whatever a junior role or whatever company would give me tokens.
00:45:31.760 --> 00:45:33.039
Tokens are like the currency.
00:45:33.119 --> 00:45:35.840
I would get paid nothing as long as you give me a pay tokens.
00:45:36.159 --> 00:45:36.719
Tokens, yeah.
00:45:36.960 --> 00:45:37.920
Pay me in tokens.
00:45:38.239 --> 00:45:38.639
Honestly.
00:45:39.039 --> 00:45:43.199
And then just like build, and I'm I mean, I am doing this now in the time that I do have.
00:45:43.440 --> 00:45:44.800
Just build as much as possible.
00:45:44.960 --> 00:45:50.559
I think the lesson I keep learning over and over is like whatever idea I come up with, make it more ambitious.
00:45:50.639 --> 00:45:56.159
Like, I've never in my life done things like build like an eval harness, and like I'm now just being like, right, let's just try.
00:45:56.320 --> 00:45:58.639
I'm just gonna plan it out with the agent, I'm just gonna attempt.
00:45:58.719 --> 00:46:00.960
And in attempting, I learn so much, right?
00:46:01.039 --> 00:46:05.599
And in failing and doing it for the fourth time over, I'm like learning so much each time.
00:46:05.760 --> 00:46:13.280
I would just get somewhere where it's like you have generous token budgets and you're just willing to put out as many prototypes as possible.
00:46:13.599 --> 00:46:17.199
Not really trying to make them that robust or not trying to turn them into products.
00:46:17.360 --> 00:46:18.000
I don't know.
00:46:18.159 --> 00:46:21.519
I think there's people who try to like take everything and be like, how do I make this into a business?
00:46:21.920 --> 00:46:24.639
Maybe that's just not the kind of itch I like to scratch.
00:46:24.880 --> 00:46:30.000
I would just be trying to make as many small usable things as possible in order to learn.
00:46:30.239 --> 00:46:32.639
It just introduces you to the full spectrum of engineering.
00:46:32.719 --> 00:46:37.840
Like I've I always had to stay very narrow on front-end and design because there's so much to learn outside of that.
00:46:38.000 --> 00:46:41.519
But now I can just be like, well, let's learn about what database I'm gonna stand up.
00:46:41.599 --> 00:46:43.519
Like, let's learn about how I'm gonna deploy this.
00:46:43.599 --> 00:46:46.400
Like, I'm the I'm coming at it from the complete opposite angle, actually.
00:46:46.480 --> 00:46:49.840
Because I uh yeah, my background is much, much more on the back-end engineering.
00:46:50.000 --> 00:46:52.639
So and I was always terrible at design and front end.
00:46:52.880 --> 00:46:53.360
Now's the chance.
00:46:53.599 --> 00:46:59.599
All of a sudden, I I'm I'm like, oh
my gosh, so I'm I'm I just realised I'm a front-end designer and I'm a I'm a front-end engineer now.
00:46:59.920 --> 00:47:01.599
Um and it just unlocks, right?
00:47:02.000 --> 00:47:12.400
Now to ask the question which I hate to get asked, but I love to ask other people, uh, which is the the getting the crystal ball out and trying to foretell how how the future will go.
00:47:12.639 --> 00:47:20.719
In a few years, let's say, because I hate you know, years ago I would have said 10 years, but uh in a few years, let's say let's say let's say six or seven years.
00:47:20.960 --> 00:47:21.760
Okay, okay, hard.
00:47:21.840 --> 00:47:21.920
Okay.
00:47:22.559 --> 00:47:25.519
How do you see that role that you just mentioned?
00:47:25.679 --> 00:47:25.920
Yeah.
00:47:26.239 --> 00:47:35.360
Um, I guess, not the on entrepreneurial, but the the the creative, the the rapid application developer, the trying to trying to build those types of applications.
00:47:35.519 --> 00:47:40.719
How do you see that standing up to the test of time of just five, six years?
00:47:41.519 --> 00:47:56.559
I would hope I would hope in the process of building lots and lots of stuff and being ambitious because agents allow you to, you would learn enough that you should be becoming very powerful with agents in a way that I think those of us inside the AI bubble forget how slow technical adoption is outside of it.
00:47:57.199 --> 00:47:59.519
Um like because we're going at warp speed.
00:48:00.000 --> 00:48:09.039
But if you go talk to most people around, they're like, basically no chat GPT, like now they're hearing a lot about the AI safety concerns, like like the bubble, like things are filtering out of the bubble.
00:48:09.280 --> 00:48:18.159
Um, but still I feel like if you got started early and were very ambitious and really understood the capabilities of what models can teach you and allow you to do, you're still way ahead of everyone else.
00:48:18.400 --> 00:48:26.159
I think it's not like those people would become that their skills would become like useless because the models advance so much.
00:48:26.320 --> 00:48:27.760
I don't see it going that direction.
00:48:28.079 --> 00:48:36.559
I don't think at any point models will be able to like read human society and culture and like make decisions about what the right thing to build is.
00:48:36.639 --> 00:48:38.000
I don't know if it would ever get there.
00:48:38.159 --> 00:48:39.679
Not in the way a human really could.
00:48:39.920 --> 00:48:44.719
So it's more like a human who can like look around them and say, What problems can I solve in the world?
00:48:44.880 --> 00:48:48.639
is always really gonna have something valuable that the agents can never have.
00:48:48.960 --> 00:48:51.599
Why do why don't you think an agent will be able to make a good decision on that?
00:48:51.840 --> 00:48:53.679
I don't think they have access to the right inputs.
00:48:53.920 --> 00:48:54.079
Okay.
00:48:54.559 --> 00:49:01.519
I feel like they are not part of a culture and a society in a way where they can see what the problems are and also the fact they don't have physical embodiment.
00:49:01.760 --> 00:49:05.840
I know that's coming people say it's coming soon with robots and cameras and stuff.
00:49:06.079 --> 00:49:19.760
But even so, having a robot present in a social situation changes it in a way that like they I don't know, they just like can't go out in the world and be part of companies or or events or social groups in a way where they would see the same social problems we would.
00:49:20.079 --> 00:49:24.800
So do you think it's not a it's not so much a model capability, it's a it's a it's a data problem.
00:49:25.039 --> 00:49:26.480
Yeah, yeah, they don't have the right data.
00:49:26.559 --> 00:49:27.760
And I don't think you could ever scrape it.
00:49:28.079 --> 00:49:32.400
How how would you scrape and you just think of like what are really successful businesses?
00:49:32.559 --> 00:49:37.920
It's like they understood there was an unsolved problem for a group of people in a certain context.
00:49:38.239 --> 00:49:56.719
I don't know how you get the data of like being in a certain place and realizing like, oh, these people really need like for a simple version, like there's no bakery on this street where lots of people would maybe buy from it, or like I see there's some problem where like uh people don't have access to to like water in this town, like that even that level.
00:49:57.119 --> 00:50:01.599
Um you need a human on the ground with social connections.
00:50:01.679 --> 00:50:06.400
If you think of like how sales works, it's all like social relationships, it's all like who knows who.
00:50:06.480 --> 00:50:08.559
It's like who who convinces who to buy something.
00:50:09.360 --> 00:50:17.920
Models I even though they might get at social engineering, they're never gonna have the social relationships you need to actually build a real business, like solve an actual problem for other humans.
00:50:18.239 --> 00:50:21.119
Um, that's the piece that models can't replace.
00:50:21.360 --> 00:50:29.199
But a human with that context, with like I've spotted a problem, I understand how I might solve it, I now have can leverage agents to actually solve that problem.
00:50:29.360 --> 00:50:32.880
That's that's actually gonna be someone who has a lot of power in the world.
00:50:33.119 --> 00:50:38.079
And power in a good way, not in a like get all the money way, but in a like solve problems for people sort of power.
00:50:38.480 --> 00:50:38.719
Yeah.
00:50:39.280 --> 00:50:40.639
Maggie, we've we're at time.
00:50:40.800 --> 00:50:47.119
Um I wonder if we can get nine technical people to to record uh a podcast episode in five minutes.
00:50:48.000 --> 00:50:50.480
Just to just to wrap up a nice, nice little ending thought.
00:50:50.800 --> 00:50:52.800
Um Maggie, this has been really good fun.
00:50:52.880 --> 00:50:56.639
I really, really appreciate your time um and and insights as well and wisdom.
00:50:57.039 --> 00:51:02.000
Where can our listeners go to learn more about Chopin as well as uh GitHub Next?
00:51:02.320 --> 00:51:02.480
Sure.
00:51:02.559 --> 00:51:08.559
So Github Next.com, we have a little write-up on Chopin, just like a very early preview of what we're working on, and we'll put more updates on there.
00:51:08.719 --> 00:51:16.079
Uh and then I have a website, Maggieapleton.com, where I put up writing and research and everything I'm working on goes on either the GitHub Next site or my personal site.
00:51:16.320 --> 00:51:16.559
Amazing.
00:51:16.639 --> 00:51:20.079
And where's where's where should people listen on socials for for for you?
00:51:20.159 --> 00:51:22.079
Or are you are you not a I'm on X.
00:51:22.400 --> 00:51:22.719
You're on X.
00:51:23.119 --> 00:51:23.760
I'm I live on X.
00:51:24.400 --> 00:51:24.960
Let's call it Twitter.
00:51:25.840 --> 00:51:26.239
It is Twitter.
00:51:26.480 --> 00:51:27.599
It was it was Twitter from the beginning.
00:51:27.760 --> 00:51:28.000
Yeah.
00:51:28.079 --> 00:51:28.639
I live on Twitter.
00:51:28.719 --> 00:51:29.280
I love Twitter.
00:51:29.519 --> 00:51:34.639
I had like a dip, but it's still the best place to l like AI news, AI people.
00:51:34.800 --> 00:51:35.280
It's great.
00:51:35.679 --> 00:51:36.079
Amazing.
00:51:36.239 --> 00:51:40.000
Maggie, thank you very much for for joining us today, and it's been a pleasure to chat with you.
00:51:40.159 --> 00:51:40.719
Yeah, thanks so much.
00:51:41.360 --> 00:51:43.440
And I hope you enjoyed that uh listening as well.
00:51:43.599 --> 00:51:45.119
And thank you very much for tuning in.
00:51:45.360 --> 00:51:45.920
See you next time.
00:51:46.079 --> 00:51:46.960
Bye for now.
00:51:47.519 --> 00:51:52.400
The AI Native Dev is brought to you by Tesla, the package manager for skills and context.
00:51:52.480 --> 00:51:55.440
Your hosts are Guy Pajani, and me, Simon Maple.
00:51:55.599 --> 00:51:57.119
Our producer is Tom Dowler.
00:51:57.440 --> 00:52:00.639
The AI Native Dev is not just a podcast, it's a community.
00:52:00.719 --> 00:52:04.239
And we host monthly meetups at the Tesla offices in central London.
00:52:04.400 --> 00:52:10.079
Visit Tesla.io, forward slash community, turn forward at the