speaker-0: at a point beyond people reading every line of code. And I think a lot of companies either are at that point or are very close to that point. And so it's almost how do you square those together I think is really tough. And, you know, this is my personal opinion from what I've observed. I just think the role of engineering is changing.
speaker-1: you So today I'm really excited to have Michael Tweed. Michael Tweed is the Principal Software Engineer at Skyscanner. And we've chatted before and it was a really interesting conversation. We've talked about AI adoption. And so I thought I'd get you back today and to join the podcast and we'll kind of dig into what's going on at Skyscanner with the AI adoption, right? I think you're kind of leading that from a DevEx point of view. Is that right?
speaker-0: Yeah, that's that's right. ⁓ so I've started started doing this over the last couple of years. I've started in a mobile engineering background for most of my career at Skyscanner that moved into mobile developer experience, sort of mobile platform. And then about two, three years ago, as we saw the push for AI, I started moving there. So now working across all our platforms on the AI adoption, what that looks like and how we work at Skyscanner.
speaker-1: Amazing. Amazing. And I remember you telling me it's actually quite a big development organization. Can you tell us a bit about the organization?
speaker-0: Yeah, so we have sort of a few hundred engineers, ⁓ you know, I think around six, seven hundred in total, and O C folks alongside that product design, for example. ⁓ so we're quite engineering heavy as an org. And so yeah, rolling out at that kind of scale's been, you know, very interesting to say the least, as I'm sure we'll get onto.
speaker-1: Yeah. Yeah. I think that's really interesting. So, ⁓ I guess let's dive straight into it. Can you tell us a bit about what was your approach? And I guess, when did you start? You have you started with AI? Have you started with agents and how have you kind of approached the rollout?
speaker-0: Yeah, so I think it's really interesting. One of the things that was quite scary to me when I found ⁓ one of the Slack channels internally was that we've been working on this for sort of four years now, which w seems I know, which seems forever if you think how fast it's moving. And it was really interesting, you know, it's like when the very first AI tools were being released and it was sort of the glorified autocomplete and it was, you know At first it was just like, is this something? Is this going to be something? How do we want to work with it? And you know, like very small groups of testers and always need to look four years on how everything has gone from there is you know, it's ⁓ yeah, it's very exciting to see. But, you know, I think yeah, what as I said, we started, we probably had at least a year, probably a bit longer in that sort of testing things out, small groups, you know, okay, this is something, you know, I like to think back to when it was some of ⁓ you know, the early product like GitHub Copilot, for example, I think was one of the first to market ⁓ with that tooling and what that mean meant. ⁓ and then, you know, I would say we really saw the shift change probably at this point about 18 months ago, maybe a little longer, you know, as the models became a lot more advanced, a lot of the labs, OpenAI, Anthropic, and others were sort of releasing and coming to market. And that was when we started to take it more seriously as Okay, this isn't just something that, you know, engineers can use as just another IDE plugin or something to go faster, but this is something that can really accelerate how we write code. ⁓ and so, you know, it's sort of been on that journey on how we move that from, you know, local individual developer workflows to making that something, you know, where things can be more shared, you know, more centralized enablement where I've got to, and now continuing to look like how we scale that across our S DLC ⁓ and how we, you know, really change how we're building internally at Sky's kind of
speaker-1: Yeah, it's interesting, isn't it? think, you know, the AI assistant phase that most people kind of went through was relatively low risk because it was kind of an assistant that was doing auto complete effectively. Now we're talking about the agents making much more important decisions. And, ⁓ and, and it seems to be, you know, so showing those signs are slowing down at the moment. Right. I don't know what you're seeing, but for me, it just seems to keep getting better and the implication is larger all the time. Right.
speaker-0: Yeah, exactly. You know, like I think we've kind of crossed that threshold. I know it's talked about a lot in industry, but you know, I think we speak a lot of verifying AI generated code or the output. So to say I think that's gone. You know, we we know that people aren't reading every single line of agent generated code right now, whether that's the engineers who are writing the code or not writing, you know, prompting the code and then pushing it or the engineers who are then reviewing that. You said I think we've crossed that threshold on it's here, it's there, it's in our code bases. I I don't think we're going to go back.
speaker-1: So what does your tooling stack look like at the moment? It's what kind of agents, kind of models are you using? I guess it's fairly diverse across that size of organization.
speaker-0: Exactly. So so I think that's also one of the things that we tried, you know, it like I think as we've matured with our developer experience as well. So for a time we were just on sort of quite standard tooling, you know, some assorted market leading and and just going with what models were out there. ⁓ what we, you know, have moved to more recently is trying to get a little bit more diversity in those models, thinking from a Not just a cost efficiency perspective, but also matching what's right for the task. And I think that's quite a natural, you know, we've seen that. It's like we ramp up adoption, you know, we go ahead, we see some pretty big numbers, and it's like maturing of as we said, now we've realized that this is here to stay. It, you know, it's not something that we trialed for a few months, decided to pull back on. You know, we're seeing the benefits. ⁓ so we're trying to do more, like we have harnesses enabled internally, but doing quite a lot of customization. you know, rollouts for engineers where we can have, you know, even sort of a basic things like centralized MCP servers for everybody, where we have that ⁓ you know, like skills marketplace. That's been something that was huge when we launched that last year, ⁓ getting skills in place where everybody can contribute, share, you know, run workflows. And then we started looking at, you know, the next step is, as I said, sort of phase one was getting engineers to write. individual workflows agents. Stage two was these repeatable skills, sharing a lot across Vorg, centralized configs where we can do a lot of governance, a lot of cool things on standardization and really like making that addition. And now going beyond that, it's also how do we scale beyond just the individual engineers, machines, you know, what can we do in the cloud? How can we move sessions there? How do we make things more collaborative? And yeah, that's we're exploring that right now, but yeah, again, kinda Interesting journey.
speaker-1: Yeah. Yeah. So, yeah, I mean, the send any centralized config, I think is really interesting, right? Cause a lot of organizations obviously starting off with just individual code codecs, et cetera, without really thinking a lot. So how have you approached that and what kind of things have you started kind of doing first skills?
speaker-0: So I think what was interesting is we tried to go very light touch at first because it's moving so quickly there's so much we don't know. You know, it's impossible even for myself in the space to to sit down and say, This is how we should be doing AI at skyscanner. This is how people should be using harnesses because it's it's impossible to keep up. ⁓ and as you said, different models work with different harnesses and different skills work in different ways. But what we also did, which we learned into, was we And have founded an AI champions network internally. So we have like a lot of departments, a lot of squads. And, you know, we know that we have lots of very, very talented engineers at Skyscanner. and you know, we have engineers not in our developer experience where I'm working, but they'd be going ahead and they'd be sharing in Slack or other tools and saying, Look at this amazing thing that I've built, or you know, look how I've built this workflow or these loops. And so we built this network to bring those people together. You were really passionate. And then that's where we started to hear the feedback. And so for example, that was one of the big things was when we started hearing them saying, hey, I've got these skills that, you know, basically I'm copying, we're copying pasting markdown files in my team. Like we're sending markdown files over Slack to each other to run. It's like, okay, that needs a bit prop, you know, a bit of proper support. That's not something that's going to scale. ⁓ and with the config as well, You know, also gives us options around like helping to put in what models we make available and you know, what the default model should be, context windows, we've even, you know, even comes down to some pretty cool things we've done with like custom status bars in the tools, for example, which we can ship where you know folks can install it. It shows you in real time your token usage and your cost, which might not sound huge, but it's like the little things that add up, kind of making this a consistent experience. And also gives us, you know, this sort of history of the changes that we're making, you know, monitoring over time. If we want to ever be able to look back and say, Did performance change on this date and see, well, was there anything we shipped? Did we change with default models? You know, what does that look like, for example?
speaker-1: We'll see a lot of concern about token costs and token usage now, but actually just having given transparency to the developers on what's what it's spending on, what it's not spending on. And, you do hear a lot of stories of people developing skills and then just realizing they're not actually getting that much benefit, but they're churning a lot of tokens with the skills. So do you have like a centralized repository or, do you kind of benchmark the skills at all? Do you do any evals on them or anything?
speaker-0: Yeah. So we've we you know, we've started Metronic to make sure we have the valves on the skills. So we actually have one of the the teams internally we have have been working on that on evals and which we also are doing, like any anything AI we ship to our customers, for example. We know we need evals ⁓ running on those and s we you know, adjusting those to use internally. And I think it's again it's like the mindset shift of well, at the end of the day a skill is a markdown file. And so you can put anything in there, but as you said, to get it efficient, you know, you want you need to run it a certain way. You it's like funny, you almost need to treat it as code, the same way that you would have optimized code before. And so trying to get in like an evals first mindset. And then we've got very meta in that we have a skill that you can use for writing a skill with evals, which is fun, you know, to guide guide through that. ⁓ and it's similar with benchmarking, you know, that's something You know, we're looking to invest more in as well as we're seeing more models come to market. And let's say for more providers as well. You know, up until now we've obviously had the Frontier Labs, which everybody knows about, but then their model families, but as we're seeing more options come into the market and we're deciding which ones do we enable, which ones do we want, you know, we're trying to make sure we have ⁓ sort of internal benchmarking and we're working on how can we do that with our tech stack as well. ⁓ to sort of think You know, there's industry benchmarks and they often use certain libraries, but okay, we use this tech stack at skyscanner. So which one's actually going to work best for us?
speaker-1: I think that's really important at this scale, right? Because it feels like the models are really just playing towards those benchmarks and the reality of what they're doing on your code in your organization with your tools could be really different. do you see, have you ended up through this kind of benchmarking getting some surprising results for you? Did certain models work that you weren't expecting to?
speaker-0: Yeah, I think like one of the things we've seen quite recently, ⁓ which has been interesting, is obviously like the thinking modes that are coming on, the newer models. But what we've seen is we're getting better results at sort of higher thinking levels on I guess cheaper or less intelligent models than we are for ⁓ sort of higher models. And it's like again, it's like really interesting. ⁓ so you know, like GPT models, okay, Lunar on Max is actually performing better and more efficient on some tasks than like GPT five point six soul on a lower thinking level. And it's like all these things that you don't really think about and it and it adds another dynamic into the mix as well. Yeah, because now you're not just comparing models, you're comparing thinking modes and it goes very it can yeah it can get very confusing to track it all.
speaker-1: Yeah. And then potentially bringing different skills or tools in as well, which change, you know, I've, I've, from my own benchmarking, seen that different models will behave with different tools in very different ways. Right. Some will understand the tool and use it. And some will kind of kind of just not really get it or just use, go through the motions of using it without really benefiting from it. I think it's interesting. You said that about Luke, you know, Luna and Sol and things. I've actually found, I did some benchmarking on Sol with like. the low thinking mode and the X high to see what you've got. And yeah, you're on my evals. was getting pass rates on both because it's such a good model. Often the higher thinking modes, they kind of explore a bit deeper and you just occasionally get something that you know would help with quality, but it's really hard to quantify the value of that, you know.
speaker-0: Exactly. Yeah. And it's like I think we're really starting to see, you know, the differences like the higher levels. I mean, it it might sound quite obvious, but I do think for a while it was, you know, AI was still such a new thing, but almost seemed so advanced compared to where it was that you couldn't always tell the difference. But we've had some quite funny cases where, you know, the way we do model routing internally, we have fallback models if a if a configuration isn't that. And like we've had times when something's broken and then we very quickly get our engineers saying, definitely not working as expected, definitely taking longer than okay, yeah, it's on a cheaper model. Like, you know, the differences are are very, very clear now.
speaker-1: So the, engineers able to choose to they have ⁓ a set of options that are kind of recommended by you for different types of tasks.
speaker-0: Yeah. Yeah. So what so we're trying to be quite, you know, well, we do trust our engineers so they have access to to all the models, all the thinking modes we have enabled. We try to set sensible defaults, which is also just so that engineers themselves, you know, we I mean, I've been very guilty myself to put a really basic query. I'm not a ⁓ you know, like yeah, I put it on sole max thinking because I was solving a challenging problem earlier and then my one basic queries now, you know, cost me a lot more than it should. So we're trying to have some sort of sensible defaults in there, but we do trust our engineers to go pick what's right for them. But then we have like internal dashboards where you can track your spends, you can get alerted. And and as I said, that's as much for the transparency. So engineers can go in and see. And, you know, again, it's like that's where we've got some really interesting I think it's been one of the most interesting shifts for me, which is not a bad thing at all. But I you know, I've worked in Engineering for a long time, lots of companies, and one of the biggest complaints I've heard over the years, ⁓ and I've been guilty of it myself, is the whole why aren't we paying for this tool? You know, if we just paid more for this, it could make or this license, it would make our life so much easier. And with AI now it's almost the first time I'm seeing engineers like looking at it and saying, Why did we pay so much for this thing? You know, I wrote one query and it cost me this many dollars. No, this doesn't seem right. I should be using a cheaper model. And it's, you know, it's a great thing, of course. And it's like great that people are paying attention and saying, hey, you know, I used these models, I can see this difference because obviously it's not just the cost. It's like the token efficiency comes into it. So it's not just the sticker price. But it is it's like a really I think it's been a really nice like cultural mindset for us internally at Sky Scanner is you know, everybody knows you can go and use the tools, but everybody's trying to be sensible with them as well.
speaker-1: Yeah. Yeah. It's really interesting as well. think, you know, the, where the engineer is keeping more control over the process and specifying more, more precisely, could then get away with much cheaper models because they're basically as following the engineers instructions. But if the, if you give the, the agent, a very open query, you often really need to do the, the, the higher reasoning modes to really explore that crew properly. That probably leads us a bit into the kind of process. So. Could you try and sort of take me through, guess it's not general, right? And I'm sure different teams are doing different things, but I kind of typical process at Skyscanner, you know, starting with requirements and what sort of level of kind of specifications are you kind of doing and finding working with agents?
speaker-0: Yeah. So it's like again, we're with not being very prescriptive across teams. So different teams can do it in different ways. What we try and do is like to you know, at least have consistency at the repo level. So everybody contributing to a certain repo as a certain standard. We have done quite a lot of experimentation ⁓ and like rollouts of S D D spec driven development frameworks. So we have compared a few as well, like again, not being prescriptive on which to use, but we did say we want teams to try this out. Mm. I think this was particularly, you know, towards the beginning of this year, you know, when we were seeing that a lot of the models, you know, needed that extra guidance, as you said. So working through that workflow. ⁓ and so we've got, you know, it's quite interesting. It's yeah, like instead of having traditional Jira tickets that have the requirements, you now have specs. And, you know, we're still again, like I think we had good results on that. overall. So it's like a lot of folks who still use it. It's not required because it's the same as like what you used to use a spec for, you can now send the soul and it can one shot it, you know, without that thing. But I think that's also really useful for sort of keeping the history. Like we have some some repositories now that have honestly been completely written by agents from the start, out of humans prompting agents. And we've got this whole library of specs that for every change because we've enforced that by default. And like the workflow is you raise a spec, humans, other engineers, you go review the spec. Does it make sense? And then once that's approved, then you can go over and apply and do it. And then almost like quite funny, you're checking the spec, you're not checking the code. Because you trust that it yeah. so I'd say that's kind of the the most common workflow we have. You know, I know that especially when it comes to like pull request reviews, for example, like I think that was actually one of our early big AI push is was to get AI reviews on all about PRs, for example. And we have teams who are running like we have kind of a base level review that's running across everything. But then a lot of teams have gone out and ridden custom agents that can do custom reviews, you know, based on their standards or what they're looking for, sort of validations there as well. ⁓ so again, we're not being like too prescriptive in an org level, but I'd say I don't want to say a hundred percent, but I'd say like the vast majority of our engineers we know are using, you know, these tools day to day to generate specs, write the code, review the code and using it as a part of their workflow.
speaker-1: Yeah, I think the, the, the spec thing is really interesting, right? Because especially if you're paying less attention to the code, actually making for the spec is the spec is right. And it makes sense. And the fact you have the, you have the kind of BDD style kind of requirements and then the design thing as well to kind of reference is really interesting. think you said it would be interesting for the, you know, the developers to look back on the history of those specs, but also for the agent, right? I guess the agent can also dig into that history and say, this feature was added. This design decision was specifically made kind of thing. Have you, as far as, you know, the, the code architecture, have you kind of done anything deliberate around that? Right. Cause I think if, I think we all kind of know if you just leave the agent to it, will tend to do the simplest thing, right? Not necessarily the right thing for the future.
speaker-0: Yeah, and I think this is where we're kinda having a bit of an internal debate as well. 'Cause it's like the two schools of four. So I guess what we are doing is we're trying to make sure we have standards in place. So we have, you know, agents MD files are sort of typical things that can shape the agent output, ensuring this. ⁓ but then we have what we call internally our production standards, so saying how we build software. Yeah you know, making sure and just I mean, these have existed for years and years at Sky Scanner. You know, and we try to make sure they're all measurable. So it could be as something as like, well, for web frameworks we use ⁓ this, this, and this. And we do not use this. And and we can sort of measure that and we have measures in place. We use an IDP internal developer portal that has all of this centralized. Okay. And we make sure that all agents have access to that. So it has access to this context, it has access to agents MD. But then we do end up in like the You know quite good questions, but we have had engineers say, Well, why shouldn't we trust the agent? If we ask the agent to build an architecture, and then, you know, one argument is, well, that's not how we do things here, and it's well, if that's what it's suggesting, it obviously has more training data on that architecture than what we use. So wouldn't we actually be better following what it knows best from its training data? It's like it's a really good point. You know, something to think about. And You know, I think that's where, you know, I think we kind of see like the more smaller projects in internal facing tools, to be honest, it's like, yeah, like one of the benefits of AI is being able to ship quickly, you know, go from idea to prototype to something working in incredibly quick space of time. And as long as we're providing sort of some of those production standards, those guidelines just to make sure it's not, you know, going completely off track. And obviously a lot of that is sort of security considerations as well. We know, okay, we don't use this because it has these potential vulnerabilities or something. Then I think we can be quite loose on that. But then when we have obviously our big projects, sort of our production code that has been, you know, powers all of skyscanner and and all of the traveler facing things, of course we want to stay a lot more consistent. And like there's very deliberate architectural choices that we've made over the years that, you know, we're not going to just rip down because of one agent because again, maybe the next model that comes along will have been trained on something different. And you know, so w we want to try and stay consistent there.
speaker-1: It's interesting, it? mean, if you do want a consistent standard across and you've got that documentation in the past, that documentation would obviously quickly become stale because it would become embedded in the engineers and they wouldn't read it and they wouldn't update it. Right. But now if the agents are using it, there's a huge emphasis on keeping that accurate and up to date and efficient to read and things. So if you've got any kind of feedback loops where you're updating the standards more now, because they're a bit more critical.
speaker-0: Yeah, we're trying to. So so I think yes in terms of how we're doing those feedback groups, so is ⁓ we we try to make things measurable and then it's quite funny because it's you know, it's again solving the problem in a right way. ⁓ or like a fun way I have engineers saying, ⁓ we're getting all these warnings that our code isn't compliant because, you know, we're not using this minimum version of this. So we haven't migrated from this internal library to this one. It's like it's so annoying. Can I just point my agent at it? It hasn't, you has an MCP server where we just say go fix the issues. It's like, yes, that's great. That's what we want you to do. You know, like that's the kind of thing where excellent you know, it's reducing that toil. But then as you said, specifically on docs, you know, like we have found that and you mentioned earlier around like specs for the context of the agent or why changes were made. Like that's ⁓ obviously a huge thing. But we found with anything, you know, like we tried at first. Okay, can we just give our agents access to, you know, confluence documents, you know, GitHub wikis, for example. And of course, it can't tell, ⁓ you know, this was a draft or this was only a proposal that never actually got implemented because it you know can't companies always together. And so we were finding some quite fun results, you know, I say fun, you know, quotations results. Like, why did it do this? And then it was one page from years ago that, you know, was never implemented. And so we're trying we actually are working on internally, like an internal sort of MCP server that has more curative knowledge and that has access. Yeah. But then we've also been using that, like we also used AI to run as a classifier against that to judge. So we put in a bunch of docs and it's okay, can you market? You know, for example, some of the basic things, formatting, context, but you know, also judge it against. How fresh is the page? How many contributors did it have? Did it have a lot of comments that were resolved? Because that can also show like the engagement, you know, like are the people still at the company? And again, all as inputs, not as the be all and end all, but actually help us to define, you know, okay, what are like the top pages that we know are kind of canonical that we want to document here? Because, you know, it's like I know there's always kind of the the jokes that go around on the work that we want AI to do and not to do. And I can tell you that having us as engineers trawling through documentation pages to manually judge them to defeated VI is not something we want to be doing. So, you know, it's like we've tried to automate that as much as possible, but again, trying to like help provide all of that context.
speaker-1: It's interesting, it? mean, with the agents writing the code though, more of the engineer's job is setting the standard. And I think this kind of thing that you talk about here, know, almost encoding the, those senior engineers kind of experience into something the agent can use. It's kind of, for me, that's the critical problem. That's like the big unsolved problem at the moment. What format does that look like? How do you, how do you manage that across hundreds of developers with different opinions and things? It's kind of, it will be really interesting. I'll be interested to see how that kind of, project kind of evolves because yeah, just throwing it at confluence is going to be like.
speaker-0: Yeah, doesn't work. Yeah.
speaker-1: Yeah. Cool. Cool. That's amazing. All right. So we've talked a bit about the requirement side using the specing and, obviously the getting the architecture right. And it seems fantastic that you can use this kind of MCP to pull some of that. What about the verification? I've obviously, you know, with something like sky scanner, I don't know how many users you have, right? It's, it's millions. Yeah. And it's critical, right? You make a mistake. People get on the wrong plane, I guess. so. So how do you kind of manage that? How do you control the risk?
speaker-0: Yeah. So again, it's something we're kinda actively working on. We've got some some really interesting things in flight, like control crew, but you know, I think like when we started it was very much like PR verification gates, right? You know, so going in, we've always been very strong on CI checks, CI you know, CI reviews. But as I said, first of all, it's like we said earlier, we know that humans aren't reviewing every single line of code if they ever were. Which again, not a bad thing, but we've all been there. Even if it was the person who sits next to you, thousand line PR, you're not going to read it all. And so, you know, a lot about is making sure we have the right sort of tests in place, the right verification. you know, even that's not enough. I think I maybe mentioned this last time we spoke, but we did have quite a funny one internally when we had an incident and it was You know, ⁓ but all the tests passed on CI and the test should have caught this issue, but somehow it didn't. And then looking through the logs, the agent had gone, ⁓ this test isn't passing with this change. So I'm going to edit the test to make it pass rather than fixing the code. And that went on court. ⁓ so you know, you have to look out for that. But then you know, so I think there's that that side, but then we've started to try to move that earlier because I think there's two reasons. You know, first of all running on CI can be slow. And you know, again, we do it. I do it when I code. It's like you just put it into whatever harness you're using and you just say watch this PR and you know, the build went red. And I just put in the URL and I say look at the errors and fix them. And then push it and come back. And but also, you know, that's also like not a good use of CI resources, for example. You know, if we're pushing everything there. And So we've started trying to move a lot more about locally. ⁓ but you know, a lot of and like a lot of it we'd say we always should have been local. Like you should you should not that we always do, but you should be running tests before you push. You should have linters in place that run sort of pre commit hooks. But we've tried to like we've got a lot of folks building really interesting things on like what we're calling like a genetic development loops. So in a way it's kind of a skill that can set it up but run in a certain way in the harness, but it's saying, you know, every time you make a change, it has a hook that says you need to run these checks and you need to verify that they're passing and if they're not passing to do this. And of course, it has very clear like you can only change this, you can't change your production behavior, what you're going through. And, you know, the next step, which is bigger. It's like because driven by AI, but not something that's purely because of that is, you know, how do we optimise that even further, getting quite intelligent on saying, knowing which tests to run as the agents running, if you touched this piece of code. And again, that it's like again, there's really funny loop within the loop of you can get AI to help determine which parts of the code need to be rerun or need to be validated. ⁓ but sort of that way, so it's more like the end to end you can Give it a command like, hey, build my feature to do this or implement this spec. And you have a lot of confidence before you even push it to CI that it's going to be successful. ⁓ and like we try to do, as I said, like a mix of static code analysis. So, you know, and not not saying this just because I'm on this podcast, but we do use Sonacube, for example. And and I think that's like a good independent verifier because you know, we're we're having something external. Even though we can customize the rules, it's external to Sky Scanner, so we can kind of tr trust it a little more of having that input, ⁓ but also having, you know, deterministic rules and gates in there. And then that's qualified with sort of our internal tests, linting, and all of that to kind of give us that confidence.
speaker-1: Yeah. Yeah. I think it's interesting. We've got this kind of the determinism you can do with things like static analysis and putting it into hooks gives you a certain amount of determinism, but then you sort of want to mix that with the intelligence of the, the agent as well to do the right things at the right time and not just have kind of things. guess the nice thing about things like static analysis is it's fairly fast and efficient. So it's not a huge cost anyway. Right. So that's cool. Cool. So you touched on it at the beginning, you kind of mentioned you were starting to look, think, at more autonomous kind of agents running more autonomously. sounds like maybe it's early days, but what kind of things are you thinking about there? Or is it for specific task types or things that, you know, you.
speaker-0: So yeah, I guess it's like kinda any any use case really is kinda how we started. So like we started with the challenge, which was just we have engineers who are running things on their machines. And then you reach first of all, like multiple bottlenecks, a human bottleneck, you can't pay attention to everything. There's sort of machine bottlenecks, your computer can't run everything. You know, that's like ⁓ hey, I want to run, you know. a change across multiple repositories, but now I have to have them all checked out on my machine and I don't have enough disk space or I'm, you know, agent is struggling like basic things. But then also, you know, typical laptop goes to sleep, can't keep working, or, you know, I'm going on holiday and I need my teammate to look over something. And so that's kind of how we started was okay, so we built like this agent orchestration platform, started very basic of how, you know, it doesn't support local default Yeah, like fully local to remote handoff, but you can start a session remotely, give it the prompt, give it the tools, and it's sort of grown over time where you can kind of replicate what you're doing locally. It has access to the same skills, you can give it MCP servers, like all the same loops and validations that you run. And we're seeing like a lot of really interesting use cases there, because you can add custom triggers, ⁓ whether that's every time a PR is raised, for example, or just schedule or like typical Slack responses every time someone tags it or sends a message. And so we've seen this huge, you know, difference in use cases between folks who are just using it to once a day, you know, provide a summary of all the activity on my repo. If it's a monorepo, for example, what's happened. Or you know, Slack responses, every time I tag the agent, it can answer my questions. ⁓ or you know every time I tag it on a juror ticket it can go and and implement it sort of all of that. ⁓ and so we've seen like really as I said anything you can think of that engineers will do, they're doing, which is great. Yeah. But then, you know, I think kind of the next the next thing that we are then thinking about is again how do we like scale development? ⁓ because it's it's an interesting one on how, you know, we run at scale. But thinking, as I said earlier, like I'd I'd say we've in quite a good position now. If we think of maybe your typical hierarchy of tasks, you have maybe like a ⁓ I don't know, but you have like a task and then you have stories and then you have ethics, for example, and you have like bigger and bigger work streams. And I would say right now, you know, we're very confident at running AI on like the task or story level. So you can give it still quite scoped, but you know, you can give it run end to end. Give it a spec, come back. But it's how do we move that to maybe that more like epic level where it is multi-stage? It has things that depend on it, but we also want that human in the loop. You know, it's like, okay, we're going to do this, then verify it as expected. And then once we verify, then we can do this, or you know, once we've rolled this out to one percent of traffic and verified it, we want to build upon it. And it's for that epic, you know, that epic level where even what I was talking about before, these development loops become a lot more important where you almost want to just be able to point the agent to it and say, Okay, maybe you have to run a lot of steps, you need to do a lot of validation, you need to work for a while, but come back to me in several hours and just tell me, verify the first artifact, you look, you press go and continue. And sort of that's like the next stage in the journey I think we're seeing of how do we really get this value at SkyScatter? and building upon it. And and yeah, it's an issue of time because again it raises all sorts of good challenges for us to try and solve.
speaker-1: Yeah. Yeah. I mean, you're sort of beyond the point that you're verifying every line of code or even close to it at that point, right? You're really just looking at the outcomes or making sure, you know, and, and relying a lot more on the agent for the, on the verification, I guess, as well to do the job. You get a lot to verify. Interesting. Have you, I mean, guess SkyScan has been around for a little while. Have you been looking at refactoring my older legacy code or anything like that at all? have got agents working on legacy code? How have you found that? We do.
speaker-0: ⁓ And it's like that's ⁓ you know, like it it's a good challenge. And and like one of the things we have found is like, you know, sometimes especially when it's not a training data, it can be a bit of a challenge getting it set up. So, you know, even one example I found myself, so I said my background was in app engineering and early on, like agents worked fantastically on Swift. They did not play nicely with Objective C. Which was a problem when the goal was to try and refactor that Objective C into Swift as sort of some of the oddest parts of the code. But then I think like what we're seeing is as much as like as long as you're putting those deterministic outputs, you know, and I think that's important. So where we're seeing a lot of success because we do have a lot of teams using it. And, you know, for example, like when I was talking about this agent orchestration platform and scaling across repos. So within rot one agent run, because it's running remotely, you can point it to multiple repositories, which makes it great for saying, you know, I need to update this library from this to this. Yeah. And update it in all the places. Or, you know, I want to update this library, but then I also want to update all its callers so that they don't break when I push the update, for example. so we're seeing a lot of success there. But then it is as much like she said we have to trust the code. So if we're if we have very clear contracts which say before the code was doing this. And now it still needs to do exactly the same thing. Yeah. And then that's what we can give to the agent. Yeah. And then it's like, yes, and we're gonna as I said earlier, we have our production standards, so we're giving it to that so it knows what libraries it can and can't use. Then that's where we're seeing, you know, a lot of success on really accelerating what might have taken weeks or months before into a much, much, much shorter time frame.
speaker-1: Amazing. Amazing. And just, mean, we're always coming to the end, but I just want to ask about the teams. So how, how are they enjoying this, I guess, for, for beginning and, and how you bring in new, new team members in and, and, and how are they onboarding with AI? you finding, I mean, lots of people are worried that if you're not hand coding on the, on the code, you're never going to learn it well enough, but have you sort of tackled that problem of onboarding new engineers?
speaker-0: It's an interesting one. I think I mean overall I think engineers are enjoying it. I I hope so. You know, I know that there's always like different levels and obviously there's a lot of different pressures that go into that and delivery pressures and different areas and different things that can make that tougher. But you know, I think going on this journey and we try to do a lot of training internally as well and running sessions and sort of hands on workshops to get people up to speed is good. You know, I do think it is really interesting because I mean you said it Beth, if you're not reading the code, how can you help understand it? And then we also said for us guys kinda we're at a point beyond people reading every line of code. And I think a lot of companies either are at that point or are very close to that point. And so it's almost how do you square those together I think is really tough. And I think from my you know, this is my personal opinion from what I've observed, I just think the role of engineering is changing. ⁓ and like what's really interesting is for example, we spoke earlier around these loops and these validation loops and how we do it. But these are engineers who are actually writing code to perform these loops. It's like, well that's coding. Yes, yes, you know, exactly. Exactly. So you're still doing engineering things. You're not just there simply writing one line, you know, implement this feature. There's still so much that goes into it. I think just how you do it is shifting.
speaker-1: Engineering on the
speaker-0: And you know, I think it's just a kind of not even a skill sh skill set shift. It's more just like the mentality of how you think about it. And so I think that's more of what we're trying to bring our engineers on. It's not, you know, okay, you're onboarding, now you need to understand every line of code that Sky Scanner does and you need to use agents which are writing code that you're not going to read, but more this is how we're operating at Skyscanner. we're trying to make sure you have the right setup, the guardrails in place. So more of you can understand like the product mindset that we have, the outputs that we want to see, like our ways of working, for example, like, you know, I'd say that's much more important. When do you Go and speak to, you know, your principal engineers, what decisions need to be made, how do we document them? What artifacts would we expect to see? When the product gets involved, when do we run experimentation? You know, those are all like kind of big sky scanner things that are very important to us. And as long as you have that mindset, and then, you know, of course you need to have some, you know, an understanding of the code, but I don't think you need to understand it line by line. And, you know, it's great pointing an agent again. explain to me what this class does or what this features function does. But then just a it's kind of reducing that gap when actually you can get onboarded a lot quicker by if we've set it up correctly, we have the right agents MD files, the right skills. You could you know, our goal is that you could come into a code base, whether that's from outside of SkyScout and you've just joined or you're rotating to a new team and we're reducing the barrier that you could actually ship changes on day one. Because your setup is there and and you know, the agent's all good to go.
speaker-1: It's interesting, isn't it? We've sort of taken programming out of the requirements in a sense, but actually the engineering level is now gone up. And in fact, there's more focus on architecture, requirement, definition, all those kind of, they're always interesting bits, right? You just don't have time just sort of tapping away at the keyboard coding so much anymore. I think it's a good time for engineering. So where do you see things going? I mean, it's obviously the agents get better, the model gets better, where do think you'll be in six months time?
speaker-0: I mean, if you asked me six months ago, I probably wouldn't have even known where I'd be today. So it's like, you know, but I think it it is continuing sort of on that path. I think it's you know, we're definitely going to see more work going into the cloud. And I think it's like it sounds very cliche to be like kind of removing the bottlenecks, but it's more like it, you know, there's no it's certain a lot of people, but if you're shipping code at a much faster rate, how do you validate that? And you know, I think like one of the other things Well, I would like to look into and we have started to look into, but it's almost like not just closing like the validation loop which is running internally, but what opportunities and risks are there to kind of extend that into production. For example, I've always had the feeling of if you're shipping to production, but you have incr you know, we have best in class observability, for example, that if we ship a broken change, then we can detect why it was broken, fix it, roll it out. at the right time, ⁓ then that allows us to ship a lot faster too, because we can reduce that. But, you know, then we have to be very, very careful. ⁓ you know, like we've always been very traveler first at Sky Scanner. We have very strong SLAs that we really try to adhere to internally. And it's like, how do we get that balance? Because, you know, I think we've all seen whether they've been caused by AI or not. You know, some companies that have really been struggling and it's like Yeah, being able to experiment with AI, push changes so much quicker, get things out, mean nothing if, you know, travelers go and getting error pages and you know, it's like might not mean that much to us if we're just like, ⁓ but we fixed it in ten minutes, but in that ten minutes, you know, a lot could happen. So I think that's going to be the big challenge is that's sort of reliability side as well, balancing how we get you know, get more value and with staying reliable as well. So I think that's yeah, that's where I'd like to be us exploring more in the next six months.
speaker-1: Yeah, yeah, super interesting when you start, you know, if you give the agent the observability, then the next logical step is that it could just fix it and roll out the change. But that goes wrong. ⁓
speaker-0: Yeah. Yeah, yeah. Exactly. Yeah.
speaker-1: Yeah, that's super interesting. Cool. All right. Well, that's been amazing. It's just really nice to get the insight into what's going on at Skyscanner in a kind of large scale. And it's really impressive how you kind of let the engineers kind of, you know, get their, bring their ideas, but with some level of kind of standardization and control as well. It's super interesting. So anything else you want to share before, ⁓ before we finish?
speaker-0: think so but it's a really exciting time. I think I'm still very optimistic on the role of engineers in engineering. I'm very biased because still an engineer myself and you know I try to be hands-on when I can. But you know, as I said, I think it's just yeah, everything's moving so quickly. And it's like sometimes, yeah, taking that step back, looking at what are we really doing, you know, don't need to be too in-depth of the exact latest model or the latest harness. That when we came out yesterday But yeah, I think it it is a really interesting time for the industry. I'm excited to see where it goes.
speaker-1: Yeah, me too. Cool. Amazing. Well, thank you very much. And thank you for having me. forward to hearing how it goes on in the future and maybe we'll catch up again.
speaker-0: Let's hope so. Later. Yeah.
speaker-1: you So today I'm really excited to have Michael Tweed. Michael Tweed is the Principal Software Engineer at Skyscanner. And we've chatted before and it was a really interesting conversation. We've talked about AI adoption. And so I thought I'd get you back today and to join the podcast and we'll kind of dig into what's going on at Skyscanner with the AI adoption, right? I think you're kind of leading that from a DevEx point of view. Is that right?
speaker-0: Yeah, that's that's right. ⁓ so I've started started doing this over the last couple of years. I've started in a mobile engineering background for most of my career at Skyscanner that moved into mobile developer experience, sort of mobile platform. And then about two, three years ago, as we saw the push for AI, I started moving there. So now working across all our platforms on the AI adoption, what that looks like and how we work at Skyscanner.
speaker-1: Amazing. Amazing. And I remember you telling me it's actually quite a big development organization. Can you tell us a bit about the organization?
speaker-0: Yeah, so we have sort of a few hundred engineers, ⁓ you know, I think around six, seven hundred in total, and O C folks alongside that product design, for example. ⁓ so we're quite engineering heavy as an org. And so yeah, rolling out at that kind of scale's been, you know, very interesting to say the least, as I'm sure we'll get onto.
speaker-1: Yeah. Yeah. I think that's really interesting. So, ⁓ I guess let's dive straight into it. Can you tell us a bit about what was your approach? And I guess, when did you start? You have you started with AI? Have you started with agents and how have you kind of approached the rollout?
speaker-0: Yeah, so I think it's really interesting. One of the things that was quite scary to me when I found ⁓ one of the Slack channels internally was that we've been working on this for sort of four years now, which w seems I know, which seems forever if you think how fast it's moving. And it was really interesting, you know, it's like when the very first AI tools were being released and it was sort of the glorified autocomplete and it was, you know At first it was just like, is this something? Is this going to be something? How do we want to work with it? And you know, like very small groups of testers and always need to look four years on how everything has gone from there is you know, it's ⁓ yeah, it's very exciting to see. But, you know, I think yeah, what as I said, we started, we probably had at least a year, probably a bit longer in that sort of testing things out, small groups, you know, okay, this is something, you know, I like to think back to when it was some of ⁓ you know, the early product like GitHub Copilot, for example, I think was one of the first to market ⁓ with that tooling and what that mean meant. ⁓ and then, you know, I would say we really saw the shift change probably at this point about 18 months ago, maybe a little longer, you know, as the models became a lot more advanced, a lot of the labs, OpenAI, Anthropic, and others were sort of releasing and coming to market. And that was when we started to take it more seriously as Okay, this isn't just something that, you know, engineers can use as just another IDE plugin or something to go faster, but this is something that can really accelerate how we write code. ⁓ and so, you know, it's sort of been on that journey on how we move that from, you know, local individual developer workflows to making that something, you know, where things can be more shared, you know, more centralized enablement where I've got to, and now continuing to look like how we scale that across our S DLC ⁓ and how we, you know, really change how we're building internally at Sky's kind of
speaker-1: Yeah, it's interesting, isn't it? think, you know, the AI assistant phase that most people kind of went through was relatively low risk because it was kind of an assistant that was doing auto complete effectively. Now we're talking about the agents making much more important decisions. And, ⁓ and, and it seems to be, you know, so showing those signs are slowing down at the moment. Right. I don't know what you're seeing, but for me, it just seems to keep getting better and the implication is larger all the time. Right.
speaker-0: Yeah, exactly. You know, like I think we've kind of crossed that threshold. I know it's talked about a lot in industry, but you know, I think we speak a lot of verifying AI generated code or the output. So to say I think that's gone. You know, we we know that people aren't reading every single line of agent generated code right now, whether that's the engineers who are writing the code or not writing, you know, prompting the code and then pushing it or the engineers who are then reviewing that. You said I think we've crossed that threshold on it's here, it's there, it's in our code bases. I I don't think we're going to go back.
speaker-1: So what does your tooling stack look like at the moment? It's what kind of agents, kind of models are you using? I guess it's fairly diverse across that size of organization.
speaker-0: Exactly. So so I think that's also one of the things that we tried, you know, it like I think as we've matured with our developer experience as well. So for a time we were just on sort of quite standard tooling, you know, some assorted market leading and and just going with what models were out there. ⁓ what we, you know, have moved to more recently is trying to get a little bit more diversity in those models, thinking from a Not just a cost efficiency perspective, but also matching what's right for the task. And I think that's quite a natural, you know, we've seen that. It's like we ramp up adoption, you know, we go ahead, we see some pretty big numbers, and it's like maturing of as we said, now we've realized that this is here to stay. It, you know, it's not something that we trialed for a few months, decided to pull back on. You know, we're seeing the benefits. ⁓ so we're trying to do more, like we have harnesses enabled internally, but doing quite a lot of customization. you know, rollouts for engineers where we can have, you know, even sort of a basic things like centralized MCP servers for everybody, where we have that ⁓ you know, like skills marketplace. That's been something that was huge when we launched that last year, ⁓ getting skills in place where everybody can contribute, share, you know, run workflows. And then we started looking at, you know, the next step is, as I said, sort of phase one was getting engineers to write. individual workflows agents. Stage two was these repeatable skills, sharing a lot across Vorg, centralized configs where we can do a lot of governance, a lot of cool things on standardization and really like making that addition. And now going beyond that, it's also how do we scale beyond just the individual engineers, machines, you know, what can we do in the cloud? How can we move sessions there? How do we make things more collaborative? And yeah, that's we're exploring that right now, but yeah, again, kinda Interesting journey.
speaker-1: Yeah. Yeah. So, yeah, I mean, the send any centralized config, I think is really interesting, right? Cause a lot of organizations obviously starting off with just individual code codecs, et cetera, without really thinking a lot. So how have you approached that and what kind of things have you started kind of doing first skills?
speaker-0: So I think what was interesting is we tried to go very light touch at first because it's moving so quickly there's so much we don't know. You know, it's impossible even for myself in the space to to sit down and say, This is how we should be doing AI at skyscanner. This is how people should be using harnesses because it's it's impossible to keep up. ⁓ and as you said, different models work with different harnesses and different skills work in different ways. But what we also did, which we learned into, was we And have founded an AI champions network internally. So we have like a lot of departments, a lot of squads. And, you know, we know that we have lots of very, very talented engineers at Skyscanner. and you know, we have engineers not in our developer experience where I'm working, but they'd be going ahead and they'd be sharing in Slack or other tools and saying, Look at this amazing thing that I've built, or you know, look how I've built this workflow or these loops. And so we built this network to bring those people together. You were really passionate. And then that's where we started to hear the feedback. And so for example, that was one of the big things was when we started hearing them saying, hey, I've got these skills that, you know, basically I'm copying, we're copying pasting markdown files in my team. Like we're sending markdown files over Slack to each other to run. It's like, okay, that needs a bit prop, you know, a bit of proper support. That's not something that's going to scale. ⁓ and with the config as well, You know, also gives us options around like helping to put in what models we make available and you know, what the default model should be, context windows, we've even, you know, even comes down to some pretty cool things we've done with like custom status bars in the tools, for example, which we can ship where you know folks can install it. It shows you in real time your token usage and your cost, which might not sound huge, but it's like the little things that add up, kind of making this a consistent experience. And also gives us, you know, this sort of history of the changes that we're making, you know, monitoring over time. If we want to ever be able to look back and say, Did performance change on this date and see, well, was there anything we shipped? Did we change with default models? You know, what does that look like, for example?
speaker-1: We'll see a lot of concern about token costs and token usage now, but actually just having given transparency to the developers on what's what it's spending on, what it's not spending on. And, you do hear a lot of stories of people developing skills and then just realizing they're not actually getting that much benefit, but they're churning a lot of tokens with the skills. So do you have like a centralized repository or, do you kind of benchmark the skills at all? Do you do any evals on them or anything?
speaker-0: Yeah. So we've we you know, we've started Metronic to make sure we have the valves on the skills. So we actually have one of the the teams internally we have have been working on that on evals and which we also are doing, like any anything AI we ship to our customers, for example. We know we need evals ⁓ running on those and s we you know, adjusting those to use internally. And I think it's again it's like the mindset shift of well, at the end of the day a skill is a markdown file. And so you can put anything in there, but as you said, to get it efficient, you know, you want you need to run it a certain way. You it's like funny, you almost need to treat it as code, the same way that you would have optimized code before. And so trying to get in like an evals first mindset. And then we've got very meta in that we have a skill that you can use for writing a skill with evals, which is fun, you know, to guide guide through that. ⁓ and it's similar with benchmarking, you know, that's something You know, we're looking to invest more in as well as we're seeing more models come to market. And let's say for more providers as well. You know, up until now we've obviously had the Frontier Labs, which everybody knows about, but then their model families, but as we're seeing more options come into the market and we're deciding which ones do we enable, which ones do we want, you know, we're trying to make sure we have ⁓ sort of internal benchmarking and we're working on how can we do that with our tech stack as well. ⁓ to sort of think You know, there's industry benchmarks and they often use certain libraries, but okay, we use this tech stack at skyscanner. So which one's actually going to work best for us?
speaker-1: I think that's really important at this scale, right? Because it feels like the models are really just playing towards those benchmarks and the reality of what they're doing on your code in your organization with your tools could be really different. do you see, have you ended up through this kind of benchmarking getting some surprising results for you? Did certain models work that you weren't expecting to?
speaker-0: Yeah, I think like one of the things we've seen quite recently, ⁓ which has been interesting, is obviously like the thinking modes that are coming on, the newer models. But what we've seen is we're getting better results at sort of higher thinking levels on I guess cheaper or less intelligent models than we are for ⁓ sort of higher models. And it's like again, it's like really interesting. ⁓ so you know, like GPT models, okay, Lunar on Max is actually performing better and more efficient on some tasks than like GPT five point six soul on a lower thinking level. And it's like all these things that you don't really think about and it and it adds another dynamic into the mix as well. Yeah, because now you're not just comparing models, you're comparing thinking modes and it goes very it can yeah it can get very confusing to track it all.
speaker-1: Yeah. And then potentially bringing different skills or tools in as well, which change, you know, I've, I've, from my own benchmarking, seen that different models will behave with different tools in very different ways. Right. Some will understand the tool and use it. And some will kind of kind of just not really get it or just use, go through the motions of using it without really benefiting from it. I think it's interesting. You said that about Luke, you know, Luna and Sol and things. I've actually found, I did some benchmarking on Sol with like. the low thinking mode and the X high to see what you've got. And yeah, you're on my evals. was getting pass rates on both because it's such a good model. Often the higher thinking modes, they kind of explore a bit deeper and you just occasionally get something that you know would help with quality, but it's really hard to quantify the value of that, you know.
speaker-0: Exactly. Yeah. And it's like I think we're really starting to see, you know, the differences like the higher levels. I mean, it it might sound quite obvious, but I do think for a while it was, you know, AI was still such a new thing, but almost seemed so advanced compared to where it was that you couldn't always tell the difference. But we've had some quite funny cases where, you know, the way we do model routing internally, we have fallback models if a if a configuration isn't that. And like we've had times when something's broken and then we very quickly get our engineers saying, definitely not working as expected, definitely taking longer than okay, yeah, it's on a cheaper model. Like, you know, the differences are are very, very clear now.
speaker-1: So the, engineers able to choose to they have ⁓ a set of options that are kind of recommended by you for different types of tasks.
speaker-0: Yeah. Yeah. So what so we're trying to be quite, you know, well, we do trust our engineers so they have access to to all the models, all the thinking modes we have enabled. We try to set sensible defaults, which is also just so that engineers themselves, you know, we I mean, I've been very guilty myself to put a really basic query. I'm not a ⁓ you know, like yeah, I put it on sole max thinking because I was solving a challenging problem earlier and then my one basic queries now, you know, cost me a lot more than it should. So we're trying to have some sort of sensible defaults in there, but we do trust our engineers to go pick what's right for them. But then we have like internal dashboards where you can track your spends, you can get alerted. And and as I said, that's as much for the transparency. So engineers can go in and see. And, you know, again, it's like that's where we've got some really interesting I think it's been one of the most interesting shifts for me, which is not a bad thing at all. But I you know, I've worked in Engineering for a long time, lots of companies, and one of the biggest complaints I've heard over the years, ⁓ and I've been guilty of it myself, is the whole why aren't we paying for this tool? You know, if we just paid more for this, it could make or this license, it would make our life so much easier. And with AI now it's almost the first time I'm seeing engineers like looking at it and saying, Why did we pay so much for this thing? You know, I wrote one query and it cost me this many dollars. No, this doesn't seem right. I should be using a cheaper model. And it's, you know, it's a great thing, of course. And it's like great that people are paying attention and saying, hey, you know, I used these models, I can see this difference because obviously it's not just the cost. It's like the token efficiency comes into it. So it's not just the sticker price. But it is it's like a really I think it's been a really nice like cultural mindset for us internally at Sky Scanner is you know, everybody knows you can go and use the tools, but everybody's trying to be sensible with them as well.
speaker-1: Yeah. Yeah. It's really interesting as well. think, you know, the, where the engineer is keeping more control over the process and specifying more, more precisely, could then get away with much cheaper models because they're basically as following the engineers instructions. But if the, if you give the, the agent, a very open query, you often really need to do the, the, the higher reasoning modes to really explore that crew properly. That probably leads us a bit into the kind of process. So. Could you try and sort of take me through, guess it's not general, right? And I'm sure different teams are doing different things, but I kind of typical process at Skyscanner, you know, starting with requirements and what sort of level of kind of specifications are you kind of doing and finding working with agents?
speaker-0: Yeah. So it's like again, we're with not being very prescriptive across teams. So different teams can do it in different ways. What we try and do is like to you know, at least have consistency at the repo level. So everybody contributing to a certain repo as a certain standard. We have done quite a lot of experimentation ⁓ and like rollouts of S D D spec driven development frameworks. So we have compared a few as well, like again, not being prescriptive on which to use, but we did say we want teams to try this out. Mm. I think this was particularly, you know, towards the beginning of this year, you know, when we were seeing that a lot of the models, you know, needed that extra guidance, as you said. So working through that workflow. ⁓ and so we've got, you know, it's quite interesting. It's yeah, like instead of having traditional Jira tickets that have the requirements, you now have specs. And, you know, we're still again, like I think we had good results on that. overall. So it's like a lot of folks who still use it. It's not required because it's the same as like what you used to use a spec for, you can now send the soul and it can one shot it, you know, without that thing. But I think that's also really useful for sort of keeping the history. Like we have some some repositories now that have honestly been completely written by agents from the start, out of humans prompting agents. And we've got this whole library of specs that for every change because we've enforced that by default. And like the workflow is you raise a spec, humans, other engineers, you go review the spec. Does it make sense? And then once that's approved, then you can go over and apply and do it. And then almost like quite funny, you're checking the spec, you're not checking the code. Because you trust that it yeah. so I'd say that's kind of the the most common workflow we have. You know, I know that especially when it comes to like pull request reviews, for example, like I think that was actually one of our early big AI push is was to get AI reviews on all about PRs, for example. And we have teams who are running like we have kind of a base level review that's running across everything. But then a lot of teams have gone out and ridden custom agents that can do custom reviews, you know, based on their standards or what they're looking for, sort of validations there as well. ⁓ so again, we're not being like too prescriptive in an org level, but I'd say I don't want to say a hundred percent, but I'd say like the vast majority of our engineers we know are using, you know, these tools day to day to generate specs, write the code, review the code and using it as a part of their workflow.
speaker-1: Yeah, I think the, the, the spec thing is really interesting, right? Because especially if you're paying less attention to the code, actually making for the spec is the spec is right. And it makes sense. And the fact you have the, you have the kind of BDD style kind of requirements and then the design thing as well to kind of reference is really interesting. think you said it would be interesting for the, you know, the developers to look back on the history of those specs, but also for the agent, right? I guess the agent can also dig into that history and say, this feature was added. This design decision was specifically made kind of thing. Have you, as far as, you know, the, the code architecture, have you kind of done anything deliberate around that? Right. Cause I think if, I think we all kind of know if you just leave the agent to it, will tend to do the simplest thing, right? Not necessarily the right thing for the future.
speaker-0: Yeah, and I think this is where we're kinda having a bit of an internal debate as well. 'Cause it's like the two schools of four. So I guess what we are doing is we're trying to make sure we have standards in place. So we have, you know, agents MD files are sort of typical things that can shape the agent output, ensuring this. ⁓ but then we have what we call internally our production standards, so saying how we build software. Yeah you know, making sure and just I mean, these have existed for years and years at Sky Scanner. You know, and we try to make sure they're all measurable. So it could be as something as like, well, for web frameworks we use ⁓ this, this, and this. And we do not use this. And and we can sort of measure that and we have measures in place. We use an IDP internal developer portal that has all of this centralized. Okay. And we make sure that all agents have access to that. So it has access to this context, it has access to agents MD. But then we do end up in like the You know quite good questions, but we have had engineers say, Well, why shouldn't we trust the agent? If we ask the agent to build an architecture, and then, you know, one argument is, well, that's not how we do things here, and it's well, if that's what it's suggesting, it obviously has more training data on that architecture than what we use. So wouldn't we actually be better following what it knows best from its training data? It's like it's a really good point. You know, something to think about. And You know, I think that's where, you know, I think we kind of see like the more smaller projects in internal facing tools, to be honest, it's like, yeah, like one of the benefits of AI is being able to ship quickly, you know, go from idea to prototype to something working in incredibly quick space of time. And as long as we're providing sort of some of those production standards, those guidelines just to make sure it's not, you know, going completely off track. And obviously a lot of that is sort of security considerations as well. We know, okay, we don't use this because it has these potential vulnerabilities or something. Then I think we can be quite loose on that. But then when we have obviously our big projects, sort of our production code that has been, you know, powers all of skyscanner and and all of the traveler facing things, of course we want to stay a lot more consistent. And like there's very deliberate architectural choices that we've made over the years that, you know, we're not going to just rip down because of one agent because again, maybe the next model that comes along will have been trained on something different. And you know, so w we want to try and stay consistent there.
speaker-1: It's interesting, it? mean, if you do want a consistent standard across and you've got that documentation in the past, that documentation would obviously quickly become stale because it would become embedded in the engineers and they wouldn't read it and they wouldn't update it. Right. But now if the agents are using it, there's a huge emphasis on keeping that accurate and up to date and efficient to read and things. So if you've got any kind of feedback loops where you're updating the standards more now, because they're a bit more critical.
speaker-0: Yeah, we're trying to. So so I think yes in terms of how we're doing those feedback groups, so is ⁓ we we try to make things measurable and then it's quite funny because it's you know, it's again solving the problem in a right way. ⁓ or like a fun way I have engineers saying, ⁓ we're getting all these warnings that our code isn't compliant because, you know, we're not using this minimum version of this. So we haven't migrated from this internal library to this one. It's like it's so annoying. Can I just point my agent at it? It hasn't, you has an MCP server where we just say go fix the issues. It's like, yes, that's great. That's what we want you to do. You know, like that's the kind of thing where excellent you know, it's reducing that toil. But then as you said, specifically on docs, you know, like we have found that and you mentioned earlier around like specs for the context of the agent or why changes were made. Like that's ⁓ obviously a huge thing. But we found with anything, you know, like we tried at first. Okay, can we just give our agents access to, you know, confluence documents, you know, GitHub wikis, for example. And of course, it can't tell, ⁓ you know, this was a draft or this was only a proposal that never actually got implemented because it you know can't companies always together. And so we were finding some quite fun results, you know, I say fun, you know, quotations results. Like, why did it do this? And then it was one page from years ago that, you know, was never implemented. And so we're trying we actually are working on internally, like an internal sort of MCP server that has more curative knowledge and that has access. Yeah. But then we've also been using that, like we also used AI to run as a classifier against that to judge. So we put in a bunch of docs and it's okay, can you market? You know, for example, some of the basic things, formatting, context, but you know, also judge it against. How fresh is the page? How many contributors did it have? Did it have a lot of comments that were resolved? Because that can also show like the engagement, you know, like are the people still at the company? And again, all as inputs, not as the be all and end all, but actually help us to define, you know, okay, what are like the top pages that we know are kind of canonical that we want to document here? Because, you know, it's like I know there's always kind of the the jokes that go around on the work that we want AI to do and not to do. And I can tell you that having us as engineers trawling through documentation pages to manually judge them to defeated VI is not something we want to be doing. So, you know, it's like we've tried to automate that as much as possible, but again, trying to like help provide all of that context.
speaker-1: It's interesting, it? mean, with the agents writing the code though, more of the engineer's job is setting the standard. And I think this kind of thing that you talk about here, know, almost encoding the, those senior engineers kind of experience into something the agent can use. It's kind of, for me, that's the critical problem. That's like the big unsolved problem at the moment. What format does that look like? How do you, how do you manage that across hundreds of developers with different opinions and things? It's kind of, it will be really interesting. I'll be interested to see how that kind of, project kind of evolves because yeah, just throwing it at confluence is going to be like.
speaker-0: Yeah, doesn't work. Yeah.
speaker-1: Yeah. Cool. Cool. That's amazing. All right. So we've talked a bit about the requirement side using the specing and, obviously the getting the architecture right. And it seems fantastic that you can use this kind of MCP to pull some of that. What about the verification? I've obviously, you know, with something like sky scanner, I don't know how many users you have, right? It's, it's millions. Yeah. And it's critical, right? You make a mistake. People get on the wrong plane, I guess. so. So how do you kind of manage that? How do you control the risk?
speaker-0: Yeah. So again, it's something we're kinda actively working on. We've got some some really interesting things in flight, like control crew, but you know, I think like when we started it was very much like PR verification gates, right? You know, so going in, we've always been very strong on CI checks, CI you know, CI reviews. But as I said, first of all, it's like we said earlier, we know that humans aren't reviewing every single line of code if they ever were. Which again, not a bad thing, but we've all been there. Even if it was the person who sits next to you, thousand line PR, you're not going to read it all. And so, you know, a lot about is making sure we have the right sort of tests in place, the right verification. you know, even that's not enough. I think I maybe mentioned this last time we spoke, but we did have quite a funny one internally when we had an incident and it was You know, ⁓ but all the tests passed on CI and the test should have caught this issue, but somehow it didn't. And then looking through the logs, the agent had gone, ⁓ this test isn't passing with this change. So I'm going to edit the test to make it pass rather than fixing the code. And that went on court. ⁓ so you know, you have to look out for that. But then you know, so I think there's that that side, but then we've started to try to move that earlier because I think there's two reasons. You know, first of all running on CI can be slow. And you know, again, we do it. I do it when I code. It's like you just put it into whatever harness you're using and you just say watch this PR and you know, the build went red. And I just put in the URL and I say look at the errors and fix them. And then push it and come back. And but also, you know, that's also like not a good use of CI resources, for example. You know, if we're pushing everything there. And So we've started trying to move a lot more about locally. ⁓ but you know, a lot of and like a lot of it we'd say we always should have been local. Like you should you should not that we always do, but you should be running tests before you push. You should have linters in place that run sort of pre commit hooks. But we've tried to like we've got a lot of folks building really interesting things on like what we're calling like a genetic development loops. So in a way it's kind of a skill that can set it up but run in a certain way in the harness, but it's saying, you know, every time you make a change, it has a hook that says you need to run these checks and you need to verify that they're passing and if they're not passing to do this. And of course, it has very clear like you can only change this, you can't change your production behavior, what you're going through. And, you know, the next step, which is bigger. It's like because driven by AI, but not something that's purely because of that is, you know, how do we optimise that even further, getting quite intelligent on saying, knowing which tests to run as the agents running, if you touched this piece of code. And again, that it's like again, there's really funny loop within the loop of you can get AI to help determine which parts of the code need to be rerun or need to be validated. ⁓ but sort of that way, so it's more like the end to end you can Give it a command like, hey, build my feature to do this or implement this spec. And you have a lot of confidence before you even push it to CI that it's going to be successful. ⁓ and like we try to do, as I said, like a mix of static code analysis. So, you know, and not not saying this just because I'm on this podcast, but we do use Sonacube, for example. And and I think that's like a good independent verifier because you know, we're we're having something external. Even though we can customize the rules, it's external to Sky Scanner, so we can kind of tr trust it a little more of having that input, ⁓ but also having, you know, deterministic rules and gates in there. And then that's qualified with sort of our internal tests, linting, and all of that to kind of give us that confidence.
speaker-1: Yeah. Yeah. I think it's interesting. We've got this kind of the determinism you can do with things like static analysis and putting it into hooks gives you a certain amount of determinism, but then you sort of want to mix that with the intelligence of the, the agent as well to do the right things at the right time and not just have kind of things. guess the nice thing about things like static analysis is it's fairly fast and efficient. So it's not a huge cost anyway. Right. So that's cool. Cool. So you touched on it at the beginning, you kind of mentioned you were starting to look, think, at more autonomous kind of agents running more autonomously. sounds like maybe it's early days, but what kind of things are you thinking about there? Or is it for specific task types or things that, you know, you.
speaker-0: So yeah, I guess it's like kinda any any use case really is kinda how we started. So like we started with the challenge, which was just we have engineers who are running things on their machines. And then you reach first of all, like multiple bottlenecks, a human bottleneck, you can't pay attention to everything. There's sort of machine bottlenecks, your computer can't run everything. You know, that's like ⁓ hey, I want to run, you know. a change across multiple repositories, but now I have to have them all checked out on my machine and I don't have enough disk space or I'm, you know, agent is struggling like basic things. But then also, you know, typical laptop goes to sleep, can't keep working, or, you know, I'm going on holiday and I need my teammate to look over something. And so that's kind of how we started was okay, so we built like this agent orchestration platform, started very basic of how, you know, it doesn't support local default Yeah, like fully local to remote handoff, but you can start a session remotely, give it the prompt, give it the tools, and it's sort of grown over time where you can kind of replicate what you're doing locally. It has access to the same skills, you can give it MCP servers, like all the same loops and validations that you run. And we're seeing like a lot of really interesting use cases there, because you can add custom triggers, ⁓ whether that's every time a PR is raised, for example, or just schedule or like typical Slack responses every time someone tags it or sends a message. And so we've seen this huge, you know, difference in use cases between folks who are just using it to once a day, you know, provide a summary of all the activity on my repo. If it's a monorepo, for example, what's happened. Or you know, Slack responses, every time I tag the agent, it can answer my questions. ⁓ or you know every time I tag it on a juror ticket it can go and and implement it sort of all of that. ⁓ and so we've seen like really as I said anything you can think of that engineers will do, they're doing, which is great. Yeah. But then, you know, I think kind of the next the next thing that we are then thinking about is again how do we like scale development? ⁓ because it's it's an interesting one on how, you know, we run at scale. But thinking, as I said earlier, like I'd I'd say we've in quite a good position now. If we think of maybe your typical hierarchy of tasks, you have maybe like a ⁓ I don't know, but you have like a task and then you have stories and then you have ethics, for example, and you have like bigger and bigger work streams. And I would say right now, you know, we're very confident at running AI on like the task or story level. So you can give it still quite scoped, but you know, you can give it run end to end. Give it a spec, come back. But it's how do we move that to maybe that more like epic level where it is multi-stage? It has things that depend on it, but we also want that human in the loop. You know, it's like, okay, we're going to do this, then verify it as expected. And then once we verify, then we can do this, or you know, once we've rolled this out to one percent of traffic and verified it, we want to build upon it. And it's for that epic, you know, that epic level where even what I was talking about before, these development loops become a lot more important where you almost want to just be able to point the agent to it and say, Okay, maybe you have to run a lot of steps, you need to do a lot of validation, you need to work for a while, but come back to me in several hours and just tell me, verify the first artifact, you look, you press go and continue. And sort of that's like the next stage in the journey I think we're seeing of how do we really get this value at SkyScatter? and building upon it. And and yeah, it's an issue of time because again it raises all sorts of good challenges for us to try and solve.
speaker-1: Yeah. Yeah. I mean, you're sort of beyond the point that you're verifying every line of code or even close to it at that point, right? You're really just looking at the outcomes or making sure, you know, and, and relying a lot more on the agent for the, on the verification, I guess, as well to do the job. You get a lot to verify. Interesting. Have you, I mean, guess SkyScan has been around for a little while. Have you been looking at refactoring my older legacy code or anything like that at all? have got agents working on legacy code? How have you found that? We do.
speaker-0: ⁓ And it's like that's ⁓ you know, like it it's a good challenge. And and like one of the things we have found is like, you know, sometimes especially when it's not a training data, it can be a bit of a challenge getting it set up. So, you know, even one example I found myself, so I said my background was in app engineering and early on, like agents worked fantastically on Swift. They did not play nicely with Objective C. Which was a problem when the goal was to try and refactor that Objective C into Swift as sort of some of the oddest parts of the code. But then I think like what we're seeing is as much as like as long as you're putting those deterministic outputs, you know, and I think that's important. So where we're seeing a lot of success because we do have a lot of teams using it. And, you know, for example, like when I was talking about this agent orchestration platform and scaling across repos. So within rot one agent run, because it's running remotely, you can point it to multiple repositories, which makes it great for saying, you know, I need to update this library from this to this. Yeah. And update it in all the places. Or, you know, I want to update this library, but then I also want to update all its callers so that they don't break when I push the update, for example. so we're seeing a lot of success there. But then it is as much like she said we have to trust the code. So if we're if we have very clear contracts which say before the code was doing this. And now it still needs to do exactly the same thing. Yeah. And then that's what we can give to the agent. Yeah. And then it's like, yes, and we're gonna as I said earlier, we have our production standards, so we're giving it to that so it knows what libraries it can and can't use. Then that's where we're seeing, you know, a lot of success on really accelerating what might have taken weeks or months before into a much, much, much shorter time frame.
speaker-1: Amazing. Amazing. And just, mean, we're always coming to the end, but I just want to ask about the teams. So how, how are they enjoying this, I guess, for, for beginning and, and how you bring in new, new team members in and, and, and how are they onboarding with AI? you finding, I mean, lots of people are worried that if you're not hand coding on the, on the code, you're never going to learn it well enough, but have you sort of tackled that problem of onboarding new engineers?
speaker-0: It's an interesting one. I think I mean overall I think engineers are enjoying it. I I hope so. You know, I know that there's always like different levels and obviously there's a lot of different pressures that go into that and delivery pressures and different areas and different things that can make that tougher. But you know, I think going on this journey and we try to do a lot of training internally as well and running sessions and sort of hands on workshops to get people up to speed is good. You know, I do think it is really interesting because I mean you said it Beth, if you're not reading the code, how can you help understand it? And then we also said for us guys kinda we're at a point beyond people reading every line of code. And I think a lot of companies either are at that point or are very close to that point. And so it's almost how do you square those together I think is really tough. And I think from my you know, this is my personal opinion from what I've observed, I just think the role of engineering is changing. ⁓ and like what's really interesting is for example, we spoke earlier around these loops and these validation loops and how we do it. But these are engineers who are actually writing code to perform these loops. It's like, well that's coding. Yes, yes, you know, exactly. Exactly. So you're still doing engineering things. You're not just there simply writing one line, you know, implement this feature. There's still so much that goes into it. I think just how you do it is shifting.
speaker-1: Engineering on the
speaker-0: And you know, I think it's just a kind of not even a skill sh skill set shift. It's more just like the mentality of how you think about it. And so I think that's more of what we're trying to bring our engineers on. It's not, you know, okay, you're onboarding, now you need to understand every line of code that Sky Scanner does and you need to use agents which are writing code that you're not going to read, but more this is how we're operating at Skyscanner. we're trying to make sure you have the right setup, the guardrails in place. So more of you can understand like the product mindset that we have, the outputs that we want to see, like our ways of working, for example, like, you know, I'd say that's much more important. When do you Go and speak to, you know, your principal engineers, what decisions need to be made, how do we document them? What artifacts would we expect to see? When the product gets involved, when do we run experimentation? You know, those are all like kind of big sky scanner things that are very important to us. And as long as you have that mindset, and then, you know, of course you need to have some, you know, an understanding of the code, but I don't think you need to understand it line by line. And, you know, it's great pointing an agent again. explain to me what this class does or what this features function does. But then just a it's kind of reducing that gap when actually you can get onboarded a lot quicker by if we've set it up correctly, we have the right agents MD files, the right skills. You could you know, our goal is that you could come into a code base, whether that's from outside of SkyScout and you've just joined or you're rotating to a new team and we're reducing the barrier that you could actually ship changes on day one. Because your setup is there and and you know, the agent's all good to go.
speaker-1: It's interesting, isn't it? We've sort of taken programming out of the requirements in a sense, but actually the engineering level is now gone up. And in fact, there's more focus on architecture, requirement, definition, all those kind of, they're always interesting bits, right? You just don't have time just sort of tapping away at the keyboard coding so much anymore. I think it's a good time for engineering. So where do you see things going? I mean, it's obviously the agents get better, the model gets better, where do think you'll be in six months time?
speaker-0: I mean, if you asked me six months ago, I probably wouldn't have even known where I'd be today. So it's like, you know, but I think it it is continuing sort of on that path. I think it's you know, we're definitely going to see more work going into the cloud. And I think it's like it sounds very cliche to be like kind of removing the bottlenecks, but it's more like it, you know, there's no it's certain a lot of people, but if you're shipping code at a much faster rate, how do you validate that? And you know, I think like one of the other things Well, I would like to look into and we have started to look into, but it's almost like not just closing like the validation loop which is running internally, but what opportunities and risks are there to kind of extend that into production. For example, I've always had the feeling of if you're shipping to production, but you have incr you know, we have best in class observability, for example, that if we ship a broken change, then we can detect why it was broken, fix it, roll it out. at the right time, ⁓ then that allows us to ship a lot faster too, because we can reduce that. But, you know, then we have to be very, very careful. ⁓ you know, like we've always been very traveler first at Sky Scanner. We have very strong SLAs that we really try to adhere to internally. And it's like, how do we get that balance? Because, you know, I think we've all seen whether they've been caused by AI or not. You know, some companies that have really been struggling and it's like Yeah, being able to experiment with AI, push changes so much quicker, get things out, mean nothing if, you know, travelers go and getting error pages and you know, it's like might not mean that much to us if we're just like, ⁓ but we fixed it in ten minutes, but in that ten minutes, you know, a lot could happen. So I think that's going to be the big challenge is that's sort of reliability side as well, balancing how we get you know, get more value and with staying reliable as well. So I think that's yeah, that's where I'd like to be us exploring more in the next six months.
speaker-1: Yeah, yeah, super interesting when you start, you know, if you give the agent the observability, then the next logical step is that it could just fix it and roll out the change. But that goes wrong. ⁓
speaker-0: Yeah. Yeah, yeah. Exactly. Yeah.
speaker-1: Yeah, that's super interesting. Cool. All right. Well, that's been amazing. It's just really nice to get the insight into what's going on at Skyscanner in a kind of large scale. And it's really impressive how you kind of let the engineers kind of, you know, get their, bring their ideas, but with some level of kind of standardization and control as well. It's super interesting. So anything else you want to share before, ⁓ before we finish?
speaker-0: think so but it's a really exciting time. I think I'm still very optimistic on the role of engineers in engineering. I'm very biased because still an engineer myself and you know I try to be hands-on when I can. But you know, as I said, I think it's just yeah, everything's moving so quickly. And it's like sometimes, yeah, taking that step back, looking at what are we really doing, you know, don't need to be too in-depth of the exact latest model or the latest harness. That when we came out yesterday But yeah, I think it it is a really interesting time for the industry. I'm excited to see where it goes.
speaker-1: Yeah, me too. Cool. Amazing. Well, thank you very much. And thank you for having me. forward to hearing how it goes on in the future and maybe we'll catch up again.
speaker-0: Let's hope so. Later. Yeah.
