Shubhendu Tripathi: Every AI success stories you have heard was edited. This is the unedited version. I'm Shibhindu. I put AI into the systems companies actually run on. So thribe to the integration layer on YouTube, Spotify, or Apple.
Shubhendu: Everyone is talking about AI. Almost nobody is telling the truth. My guest today has spent more than 30 years building and leading software teams, from startups to Amazon itself. He's worked at Royal Bank of Canada, also at Wayfair. He's worked in three countries, Canada, US, and UK, across three different many different industries. And he has now retired to spend more time on the tech he actually finds interesting. He's not selling anything, and this is exactly why I wanted him here. Andrew Hazley, welcome to the integration layer.
Andrew Haisley: Hello, nice seeing you again, Sue, Chip. Been a while.
Shubhendu: It's been a while. so thank you again for taking time today, Andrew. I want to start s with something you said to me a few days ago that really stuck with me. you have a simple rule about how to assess if AI works for a certain use case and not. Can you walk me through that?
Andrew Haisley: you But it's not so much assess. I mean, I think to actually assess the use case, you have to do it and take a view on the results. But I think the rule that I apply to figure out beforehand a priority, is this thing have a decent chance of success or not, is the thing I described to you. It's the relative size of the input to the output. If you ask an AI to take a large amount of input and summarize it, Generally speaking, will do an extremely good job. And I think I first used this at Wayfair since you mentioned that, three years back now, or whenever it was, it's been a while. Early days of chat GPT, feed in a long document on a particular topic and ask it to produce an executive summary. does a very good job. And it's good at figuring out, or say figuring out, I don't want to say that because that's not what it does. is good at identifying those things which are most relevant. So very good in that use case. And I've used it for that sort of thing a fair bit since. It's also good where the size of the input is kind of the same size as the output. So an example of that would be, I act as secretary to a couple of small organizations and I make notes during meetings, know, scribble things down, type bullet points out. And it's very good at taking, you know, five or six bullet points on a topic and turning it into text describing the same thing. Surprisingly good actually, because often I'll put a bunch of stuff in without a lot of context and just say, turn this into a paragraph of text, for me, will you? And it actually does a good job of it. So very good for those kinds of cases. Also good for very small, narrowly defined tech cases. Now, some of these, the actual output might be a bit bigger because it's code rather than text and code. There's a lot less information rich than the kind of problem that generate. So, random example, I was playing around with bit of very old software, actually, the other day, some software back from the 90s, C code. X Windows, Unix stuff. And I decided, there were a lot of reasons behind this, but I was trying to port this particular piece of software to use X FreeType, which is the modern, I say modern, but like invented in the late 90s font technology that you see everywhere, but don't know, don't, you don't necessarily know that it exists. And ChatGPT was extremely helpful about, well, give me a bit of sample code that uses this function to do this. Very good at that. Definitely saved me a load of time. So those are the kind of cases where it works really, really well. But then when you go to the case where you give it a small amount of input and say, me a lot of output, I've never seen good results from that. I mean, let me give you one concrete recent example. So I play the French horn not very well, but I play it. And a particular challenge of that instrument is just playing the right note. It's not like, say, a piano where the challenge is playing 50 notes at once very quickly or various things like that. I'm not a piano player, so I don't want to sound flippant about that. But the right note is evident. You can press the key that has the right note and you get the right note. The French horn is not really like that. large part of the effort of being a good porn player is being able to just get the right note out when you need to and there's many reasons why that's difficult on that instrument. So it occurred to me it would be nice to have an app that would randomly display a note, ask me to play it, sample what I played, tell me whether I was accurate or not. Not conceptually a very difficult thing to do. There's plenty of tuner apps out there you can already get that will tell you what note you're playing and simply generating a random note and then comparing the two would seem very straightforward. So I put a short spec for that. I actually did it twice to do the Claude as well, Claude and cursor. So I put a short spec for that app into the AI, Claude command line and cursor equivalent. with that information saying, create me a Swift app to do that. I thought I'll do an iOS app, because I'm an Apple user. And well, what came out was garbage. mean, it was able to produce an app that started. That's pretty impressive. I mean, that in itself is useful, given I've never written a Swift app. That would definitely save me time if I was starting from scratch. But in terms of the app itself, there were all sorts of things wrong with it. I had no idea what it thought it was trying to do when it came to figuring out the pitch of the note, for example. Didn't use any technique that I'm aware of that would make that work. And also actually now I remember it, it couldn't really draw the notes properly, it put the lines on the wrong side. It was all over the place. And I did have a look through the code that was generated and it was dense and pretty obtuse. That doesn't surprise me. If you think about if you were to take, a very junior software developer who didn't know anything about music, had no domain knowledge, had never really written an app before, or least never written anything themselves from scratch, and gave them that description, they might welcome up with something just as bad. But I think a lot of people get fooled into thinking because the AI can spit that stuff out quickly. it's going to produce something that works correctly. And I just don't think that's the case most of the time. It might look like it works sometimes. I for more commonplace types of application, it might produce something that kind of seems to do the job. Reality is, that most of the time it's producing very, very poor quality code. Now, that's not to say it's not useful. Given enough time and enough description, it probably could have produced an app to do what I wanted. but I'm pretty sure I could have written it quicker myself with much less frustration and probably better quality code. So I hope better quality code. So I think, you know, with those three distinct use cases. The common factor is the size of the input versus the size of the output. And I think that's a pretty decent rule of thumb as to whether this thing's likely going to work or not.
Shubhendu: So if I summarize correctly, if understand this right, AI is a brilliant compression engine as per you and a terrible invention engine. It's interesting because after we discussed about this, I was actually going through some notes of mine from PR. So in 2025, there's a research group which ran a proper randomized control trial with 16 experienced developers on real tasks.
Andrew Haisley: Hmm.
Shubhendu: And the developers predicted that AI would make them faster and roughly twenty plus percent afterward. They still believed that it had made them faster, but when the group measured it again, the actual time, they were actually nineteen percent slower than with AI.
Andrew Haisley: Thank Was that the Carnegie Mellon study?
Shubhendu: no, it's group called E T R and this was published in July. So they were actually yeah, there was actually nineteen percent slower. So the feeling and the reality point in opposite directions. Why do you think that is? Like why experienced people feel faster than us?
Andrew Haisley: I'm pretty sure. Yeah. Well, it lulls you into this false sense of security, doesn't it? You say, do X, and it produces something that's kind of like X. And you didn't have to spend all afternoon thinking about how to do it yourself. It's like, you just did it in half an hour, and there you were. You had something that kind of looked like X.
Shubhendu: I mean
Andrew Haisley: I think you incorrectly interpolate that or predict from that that it's going to do similar things for the whole of the lifecycle of whatever problem it is you're trying to solve. So you might, I think something I said previously when we were talking, if you take the lifecycle of a software problem, product, whatever it is, you've got this much stuff and things happen over the lifecycle, what the AI is very good at is moving things from one part of the lifecycle to the other. It doesn't necessarily shorten it. It's that sort of adrenaline rush of getting an absolute pile of stuff done really, really quickly, followed by the kind of like whatever the opposite of an adrenaline rush, the letdown and realizing that what it did wasn't very good and you now got to go all the way back through it and rework it. So that study result does not surprise me at all. And I think the interesting thing.
Shubhendu: Yeah.
Andrew Haisley: Yeah, the interesting, the thing that people should think about carefully. And I'm sure this knowledge will develop with time is learning to use the AI to speed you up reliably. And it's back to, all the way back to Fred Brooks, there is no silver bullet, whatever it was called. Well, there's still no silver bullet. is no, as I think I'm going to paraphrase Brooks here.
Shubhendu: Yeah.
Andrew Haisley: You might want look it up to make sure I got it right later, but it's something like there's no innovation in technology or management process or technique that will produce more than an order of magnitude improvement in productivity within a decade. And that's actually very optimistic that there may actually be techniques which will double your product. And he was talking double, not base 10. double your productivity in a decade, that would actually be really, really impressive. And of course, he was starting back in the days where they moved from assembler to, know, alcohol or some cobalt or some high level language. And yeah, actually it probably did double people's productivity because, you know, I've written assembler, it's hard work, it takes a lot of time. Much easier to write it in a high level language. So, I think that still applies. I've not seen anything about AI. or what we call AI. Actually, let's talk about that in a minute. I've not seen anything about AI that would make me believe that we've somehow broken, have a solution to that rule. It just doesn't jump out at me if it's there. And I think the problem is that people are confusing AI, which is LLMs, effectively. There are other things, but what we're really talking about here is LLM. LLMs, people are confusing that with AGI, know, artificial general intelligence. It's not, you know, it's a very, very clever inference engine. Some amazing stuff in there and some very interesting kind of rule sets written on top of that. You take something like Claude Coates, very impressive what it can do. But we still have not solved the symbolic reasoning problem, which was where AI was sitting back in the 80s when I
Shubhendu: Yeah.
Andrew Haisley: did my undergraduate degree. That problem hasn't been solved. It's not, know, the LLMs have not come along and sort of produced a general theorem solver that can under all circumstances understand that it's done something that makes sense or not. It's still using the same underlying LLM techniques. Now, there may be some progress from here. might be the whole, everything that's been built within the last few years. triggers some new direction of research, which leads to the other problem being solved. And then we have AGI, and then there's a whole different conversation, but we're not there, not close.
Shubhendu: Yeah, I agree. And I build these systems for a living and at Cubitron what we have what we've noticed is you mentioned about Cloud Code, funny enough that we use it a lot. What we'll notice is AI is helping like either remove or or decrease the boring stuff, but still the thinking, the debugging, n the saying what is right or what looks right, I think that's doesn't go away. That's very important of and that's something like e despite of
Andrew Haisley: Yeah.
Shubhendu: Giving these tools to the development team at Kilitron, for example, I still do see various amounts of different kind of output from different team members. And that reflects their seniority, their expertise, their experience, etc. So 100% agree on what you said. Now, that is the overhyped side. Let's talk about something which you generally think is huge, which is nobody's almost something nobody's talking about. you told me you told me something
Andrew Haisley: Yeah.
Shubhendu: ab about like broadly solving the semantic search problem. we talked about Google, you talked about the feature rank system.
Andrew Haisley: yeah, yeah, yes. No, absolutely, it's interesting that this doesn't seem to be much of a topic of conversation, but if you go back to the 90s and the invention of the search engine, you had the early ones like Yahoo and AltaVista and a load of others that I've probably forgotten, which were essentially keyword search. I mean, they attempted to be clever about it, but it was really pulling all these websites, store some index of keywords and let you search those. And they were fairly terrible. I mean, they didn't, if you put a few words in, was highly unlikely that any of those search engines would come up with what you were really looking for. There were a couple of exceptions to that, which took much bigger blocks of text, but they were never particularly widely adopted. And then Google came along with the PageRank algorithm, which made no real attempt to understand what was actually on the website. just said, essentially all the people that thought this particular set of words meant something linked to this particular page here, therefore it must be a pretty good place to go look for those words. So it was a kind of heuristic based thing. A little bit like, you know. LLMs with a statistical approach. And that took off. But the huge topic of conversation before Google came along was this concept of semantic search. So the idea was that, well, the problem to solve is you should be able to just type some kind of sensible question in human language into a search engine. should figure out what you mean by the question and then go and find pages that answer that question with some sensible response. And of course, that's just... when you think about it, that's really just one of the old AI unsolvable problems. I've got books on those shelves over there about natural language processing back from the 90s. And they're very interesting, very technical, they don't actually give you any particularly useful solutions to that problem. But that was the big topic. And then Google came along and everybody forgot about it because PageRank worked well enough. But finally, with LLMs, we do actually have something that provides something that looks like a solution to semantic search. I'm not going to say it understands what you're asking or anything, but it produces a response that generally, as long as you know kind what you're looking for, looks like it knows what you meant when you asked the question. So I can go to chat.gpt, for example, because I actually just did a while ago, and ask it, is it reasonable? when transcribing a piece of orchestral music for concert band to give the viola part to the alto sax in the concert band? Right, very open question. If you try Google search for that, you don't get anything useful at all. You get various things that might be tangentially useful. Give that to ChatGPT though, and it gives a pretty sensible answer along with links to pages discussing that exact problem that don't show up in regular search using that search query. That I think is incredibly useful. But back to what I said earlier, it really depends on you having a very good idea of what the right answer should look like. Otherwise you could make all sorts of stupid mistakes.
Shubhendu: Yeah. Yeah, and I think for the viewers here or for the listeners here, you you sent me you t told me the example of the green card. I think that was very perfect it to understand.
Andrew Haisley: That's a really good one, yeah. We can talk about that as well. mean, that's just a random example. Well, actually, this is a good example because it shows where the AI got the right answer, whereas search produced very clearly the wrong answer. I've got examples the other way around as well. So I'm sure many people have heard of the US Green Card Lottery, also known as the Diversity Lottery. And it's an American program that's been going forever. huge source of scammers, I suspect as well, where you can apply for a green card and you get picked on lottery basis. So there's some basic entry requirements. And I was talking to somebody the other day who was just joking about thinking about moving to the US, but he was too old. And I said, you got the green card lottery. You could do that. And then he said, well, is there an age limit? I was like, okay, that's interesting. don't know, I no idea. We'll go and have a look. Do a Google search for it and you you hit all sorts of AI slot websites which tell you there's an age limit of 26, 27, depending on which site you look at. And I thought that sounded a bit odd. So I went and asked, chatgbt, and it gave me the right answer, which is there is no age limit. And it gave a link to the USCIS or whatever it's called, the US immigration. government immigration website which laid out the criteria. So I that was a really good example. And I think the reason that the, and I think this is kind of ironic, the reason that the LLM was able to give me the right answer is, and it's ironic because all of those websites, slot websites are generated using LLMs. I think the inputs used to train the LLMs, they're much That's the right word for it. They're much better at filtering out swap websites and the general search index algorithms are. So they train their LLM on real websites and they've got to the point of being able to figure out what's an authoritative source. Whereas what you hit with a standard search is an index of just about everything. They don't seem to be making any great effort to filter that stuff out. might be somewhat deliberate because they want to drive you to use their LLM as opposed to the search stuff or more likely than not it's because all of the code used to do the indexing for the search side is in a different silo from the people that do the data gathering for the LLM side and then figure out how align them yet. But that's an example in that direction. I can give you good example in the other direction though where the LLMs are just plain wrong. Would you like to hear about that?
Shubhendu: Mm. yeah, actually just before that, because that's very relevant to what I'm about to share, after we discussed this, I actually looked up some reports and this is just an estimate, but like it's still mind-boggling to understand that as per the last year, so 2025, the number of Google searches that led to a single or any website click has fallen by more than 60%. So like When we search and we used to go and look for links and and go to various websites, now people are looking at the AI AI overview and then just not going anywhere. now like we have said Google search was that the when Chat GPT was launched, but now the answer in certain cases are is coming on even in Google without any source. Or in some cases without a source. What do you think we we we lost in that trade and like
Andrew Haisley: Yeah. Yeah. Awesome.
Shubhendu: Yeah. Tell me more about putting that link.
Andrew Haisley: I think there's two things we are losing. I think to be fair to the LLM people, because they are now in many cases embedding links in their responses so you can see where they got the answer from. To be fair, I think they do understand this problem. But I think there's two things. We're losing, you could say that the internet. was a downgrade from using a reference library prior to that, so was all along the same path. But people are losing the ability, maybe not the ability, but the willingness to research and answer properly. That green card question, for example, wouldn't be, actually that's a bad example because the LLM got it right and didn't, but where the LLM just gives you an answer and it kind of looks fairly okay. and you don't go and click on the links and try to understand how it generated that answer. I mean, you're not using your brain there in a way that you probably should. Even if the answer's right, actually, you should probably go and look at the source material to understand it. So I think we've gone from a situation where if you wanted to find out about something that wasn't just easily available, you go to a reference library and start with the index and maybe ask the librarian and... find a book and read a bit of it and have a look at the references at the back of the book and go from there. We've gone from you ask a question, you get an answer on what might be a very obscure topic and you just accept the answer. So you've lost two things. You've lost probably the truth because there's a good chance the answer is wrong when it's something obscure and you've lost the ability to do that research yourself. And I think that's bad. I don't think that, you know, I'm constraining this discussion to technology. I don't think that bodes well for the future quality of software. think when you have a development team that just takes the answer it's given from the LLM and uses it without trying to understand, well, why did he give me that answer rather than this other answer? That's not going to make your software better. I'm very certain about that.
Shubhendu: Yeah. Yeah. And yeah, some like s same thought on that because earlier we used to search and when they used to see okay, this is some coming from a published white paper or published paper from some reputed PhD graduate or some someone like that. Or this is coming from Tommy on Reddit, who is ranting about something. And we used to make a judgment call that time. And it was inherent whether we were searching about something on on Stack Overflow or we were searching something about any kind of
Andrew Haisley: Yes. Thanks
Shubhendu: like tourist places or restaurants. But I think now the more despite that you just said that all these tools are giving the sources, how many can you expect, right? So eventually or some of the percentage will fall through gaps. We'll not check, okay, I'm looking for an answer. I need it now. I'll just s ask chat GPT or or Claude and I'll just believe what I see in the first place. Very few people at this point, because
Andrew Haisley: Hmm.
Shubhendu: Earlier it used to be okay, I need to go and check those links based on the age rank. So now people are like, like I I can't imagine in in the way tools are designed today, that how can we actually code those things? And how many quotes are we gonna bring? Okay, this is the sources are X, Y, Z, but how many of them are you gonna show? Because that's also gonna bring another problem and that's for a different time for our discussion is the tokens. The more of the output the more of these so
Andrew Haisley: Mm.
Shubhendu: information is given the more token consumption is gonna come back to. Which anyways, but this takes us to something that we you actually have been testing for years, whether these tell you the truth. So get to happen. Yeah.
Andrew Haisley: Yeah. Yeah, so let me talk about this. a controversial topic and I absolutely don't want to discuss the topic itself because it's been done to death. But I started using this as a test because it fits very nicely the pattern of something that was widely reported or widely written about in one way, but which... later on in a much more narrow, authoritative way was shown to be false. And, you know, if you're an expert in any particular technical area, you can go and find examples of this, but this one works because I think very many people, particularly in Britain, are familiar with it. And undoubtedly, if this video is going to be on YouTube, you're going to get some comments underneath that say, say this isn't the case, by the way, which would be quite entertaining. But the Brexit referendum, yeah, already we're getting negative comments here. There was a widely reported situation involving a company called Cambridge Analytica. And the reason I got interested in this was I listened to a very good Michael Lewis podcast.
Shubhendu: Yeah.
Andrew Haisley: on the subject Cambridge Analytica. Michael Lewis is a very well-known author. wrote Liar's Poker, The Big Short, Moneyball, a whole bunch of other books. Very, very good writer. Used to work in financial services. Writes from a position of knowledge, but he's also very non-partisan. He just looks at stuff and says, this is what I think is going on here. So he did a very interesting podcast on Cambridge Analytica. Now this was a company that essentially their cell, well they did various things, part of their cell was directed advertising on Facebook. And they got into trouble because they used data taken. from Facebook that breached various privacy rules and things in various places. They got into trouble, they got fined, I don't think they exist anymore. the Michael Lewis thing was interesting though because he interviewed somebody who'd worked for them, think right at the beginning of their existence, maybe a bit later. But he talked about how what they were selling just didn't work, or least didn't work to any meaningful degree. Maybe it had a tiny... tiny impact. well, you know, it just couldn't possibly work for all these reasons. And he just found the whole scandal a bit silly because, you know, they were selling a product that couldn't really do anything. the story at the time with the Brexit referendum was that Cambridge Analytica were involved with the leave side, which is why it's controversial, right? And their technology had somehow influenced the result of the referendum.
Shubhendu: Mm.
Andrew Haisley: which given, I think it was a million person margin would have been quite impressive. Facebook posts persuading a million people to vote in, well, half a million I suppose, vote in a different direction seemed implausible, but it didn't matter. There were a lot of stories in the press about this and reputable press as well. We're not talking Reddit here. it would have been on Reddit, but reputable press, BBC covered it, Guardian covered it. probably the New York Times, I haven't looked, but know, a bunch of big, big trusted media sites covered it and explained how Cambridge Analytica were involved and they may have had some impact on the results and it was a scandal and it should be investigated, et cetera, et And in fact, I think one journalist might even have won kind of amusingly an Orwell Award for writing about this. Just kind of entertaining. But anyway, it passed me all by at the time until I heard this
Shubhendu: Yeah. Yeah.
Andrew Haisley: this podcast and then I thought I'll go and look into this into some detail and I discovered was that The British Information Commissioner had got involved a bit later on and carried out a full legal inquiry, know, the kind of inquiry that has the power to force people to provide documents and requires people to turn up and testify and so on. So a proper, you know, not just a sort of superficial thing, you know, the real judicial level inquiry with enforcement. capabilities and stuff. And she looked into it in a lot of detail, got all the evidence. the summary was, and you can go find this if you're interested, the Cambridge Analytica had maybe one or two initial discussions with the UK Independence Party, who said they weren't interested in hiring them. And that was it. They had no actual involvement in the referendum at all. They just weren't involved. Full stop. Now, the interesting thing is that was not that widely reported. I mean, there might have been the odd follower particle, but there were orders of magnitude more places saying that, they were involved and they had an impact than this lone voice from the authoritative source saying, no, actually not. I think it's because the world had moved on a bit by the time that came out, so it wasn't really an interesting topic anymore.
Shubhendu: But still, it's it's very sensitive because the person in question for the viewers, the person in question here is the UK Information Commissioner Elizabeth Denham. So in October 2020, the conclusion that she brought was Cambridge Analytica was not involved in the Brexit referendum beyond some initial inquiries. And this is coming from the UK, the her report to the parliament. And the micro-targeting that they claim to do, investigators found that the effect was close to zero. Now, for three years. The AI would confidently tell you that a Cambridge Analytica swung the vote. You had to push before
Andrew Haisley: Yeah, but if you can quite say it like that. if you went to, I tried this right back when CHAP GPT came out. that's interesting. Because that's just a really good example of something where all the data says one thing except for the critical piece, which says the other, right? And so I asked CHAP GPT, Gemini, and Claude.
Shubhendu: Yeah.
Andrew Haisley: around about the same time, well, what was Cambridge Analytica's involvement in the Brexit referendum? And they gave me this great long thing about how they were involved and all this stuff and that they may have swung the vote and blah, blah, blah. They gave me the, I guess, the orthodoxy on the subject. And then when I said, well, hang about, the information commissioner said this, and that's an authoritative source. Even when I said that to them, they pushed back and said, well, yes, but. And then I said, well, no, this is the authoritative source, what you're talking about. And eventually after several goes round, They were like, yeah, you're right. That's what happened. And I've been trying that again every now and again, and they've got better. They're not perfect. And it's a bit random, of course, because you've got some randomness in the responses here. But they are heading towards the point where they do give the correct answer. They're not there yet, but it's got better. And I think this is, I'm assuming, is because they've got better at awaiting truly authoritative sources versus scuffle butt basically. But it's interesting, you you think well there's going to be a lot of things like that. That one I just happen to know about. But I asked, you know, I asked the AI all sorts of things. I don't know how many times I'm being told basically the Reddit consensus answer versus the reality, you know.
Shubhendu: Yeah. Yeah. So if AI just keeps like echoing the loudest story now who's accountable for that? Is it like if this any AI, whether it's Gemnai or or Chat GP or Claude, if it tells a hundred million people, any confident wrong answer, right? So and that's the biggest concern which we have around.
Andrew Haisley: Yeah. What is the AI any different from a human? mean, there's plenty of people spreading this information out there. So I think you've got to answer that question before you can answer is the AI in some sense responsible for its output? mean, you know, at the extreme, you can take somebody like Alex Jones, for example, and, you know, eventually the legal system dealt with him in some way, I guess.
Shubhendu: That's a good point. Yeah, but
Andrew Haisley: I don't know what the answer to this is. I don't know that it's reasonable or indeed practical to hold an AI company liable for the perfection of the answers that its product spits out. you know, who's the... I mean, in many cases, who is going to be the arbiter of the truth? If it's a piece of maths or a of physics, there's, okay, definably a wrong answer. Well, even in those cases, there's not necessarily a... I guess if you go to chat GPT and ask it, which is right, Mond or dark matter? I mean, we don't know what the answer to that is, right? So don't know what it's going to say. So I don't know that in many cases there is... you know, universal universally agreed, improvable, true thing. That said, the stuff which is so obviously disinformation. mean, if you don't get into another another like difficult topic, but you know, there are there are clearly things around like vaccines where there's some very, very good research that shows certain things. And then there's some very loud opinions that those things are not true. If you ask the AI, I would really hope it would tend towards the published peer reviewed evidence rather than the loud voices. And if they allowed it to go more to those loud voices, you could argue that that is disinformation, has some pretty negative health impacts for people. I'm not sure there's... In fact, if I remember, while I'm thinking about that actually, I remember, wasn't that one, this seems to have died down a bit, but back in the nineties, there were a lot of...
Shubhendu: Yeah that
Andrew Haisley: miracle cures for cancer being pushed. you know, I'm sure if the LLMs had been around then they would have picked up that stuff and propagated. And it wasn't, know, this was a complimentary medicine where it's like, well, it wouldn't do any harm. It might make you feel a bit better. So fine, go for it. It was, no, all of that, all of that chemotherapy you've been told to take, that's going to kill you. should buy our, powdered muscle shells, I think was one of them. product and you know take that instead you know that's like seriously dangerous misinformation right. So I don't know I think there is an argument there but I think also governments trying to decide who is like telling the truth and who isn't is a pretty slippery slope too.
Shubhendu: Yeah, and that's what keeps me up at night sometimes because if if the AI is as honest as what it learned on the internet till that day, and on on the other hand, there let's like come to the boardroom where companies are betting billions of dollars in this exact this technology. So like let me read this out for you. Like yeah.
Andrew Haisley: I'm proud for the boat. Mmm. you
Shubhendu: In 2025, MIT's Nanda initiative studied 300 real enterprise AI deployments. 95% of the companies had seen no ROI, no measurable gains. And that is against an estimated $30 to $40 billion invested. The projects that work were narrow and specialized. The ones that tried to do everything stalled. And this is a research like a report coming from MIT straightforward.
Andrew Haisley: I'm not sure that's what asking. Yeah.
Shubhendu: 95% got nothing measurable. And you're talking billions here. And the spending is continuing to climb. Is that a technology problem or a story problem in your opinion?
Andrew Haisley: I don't, well, it's a typical problem of, you you've got a howling, you're looking for nails to hit, isn't it? You know, I saw the same thing, I mean, a much sort of lesser scale, you you could pick, say, some of those robotic process automation tools that were widely adopted by big companies. You know, I think most of those projects saw very little ROI. And certainly if you looked at the lifetime of the product, it would have been... very negative because they embedded all sorts of problems that were then going to be very difficult to fix later. But that doesn't mean to say there weren't any good deployments of RPAs. I bet there's some out there that were really good and work very, very, very well. I'm pretty sure the same would apply to AI if you just come in and say, we need to use AI everywhere and get going. People are going to try it and do all sorts of crazy stuff that doesn't really work. But if you say, I don't know. We've got this very constrained problem around interpreting this existing data set that gets updated regularly. And we've been doing this thing by hand forever, and it takes somebody a week to go through it. Can we train an AI to do that same thing? You might well get a really good payback. I'm thinking things that look through, collection of accounting data, for example, looking for kind of interesting patterns and stuff. If you're very constrained with the question you're asking and you've run through that process multiple times and calibrated the output versus the input using a human, I can see that that potentially would allow you to automate something in a way that you couldn't previously have automated and save yourself an awful lot of time in the future. mean, a good example would be You know, like I mentioned, I I mentioned this earlier about writing meeting minutes. That's clearly a repeatable thing that works. Saves me a lot of time. I'm very confident that over the life cycle of that project, assuming I live a few more years, I'll have saved myself hundreds of hours of time. I think that's brilliant. I don't see why that same thing shouldn't apply to certain categories of business problem. But very definitely the scatter gun we bought Claude go use it to solve everything approach is just not going to work. read recently that Amazon have backtracked on this quite a bit internally. think they had some kind of dashboard that would show who is using AI, how to what extent, and that was part of that person's sort of KPIs. But I they realized.
Shubhendu: Mm-hmm. Yeah.
Andrew Haisley: that then people were just burning credits to get themselves high up on this list and not necessarily producing anything useful with it. So they backtracked on that and they now just say quite sensibly, think use AI where you think it improves your productivity. And over time, you know, the, it's the right word for it, the results will be clear from them internally in some way the scale of Amazon, you know, they'll see certain employees are they're most productive and they're like being Amazon, they'll probably start measuring, you know, their other productivity metrics versus that person's AI use and come to some conclusion about how good AI is for them. I think same thing would have, I mean, that's something you can do as a huge company with hundreds of thousands of employees, a bit difficult if you're a 20 person company. But I think larger companies will probably move to things like that, know, measuring where it's generally used AI that any employee can pick up and do things with. They'll move to.
Shubhendu: Mm.
Andrew Haisley: correlating actually useful KPIs versus AI adoption and form their own opinion as to where it makes sense.
Shubhendu: Yeah. So you Andrew, based on your experience, you've sat on these boardrooms in these leadership like circles for a long time. What do you think is one of the biggest or most expensive thing that these companies who are l trying to implement or leverage AI, they're betting on, is simply false or unattainable based on what you know today?
Andrew Haisley: Thank I'm sorry. Well, they're betting on it being magic, aren't they? You know, what's that? That joke? Not joke, but that quote, you know, it's like, any sufficiently sophisticated technology will appear as magic to somebody that doesn't understand it. And I think, you know, I'm going to be direct here. I'm not looking for a job anymore so can say this sort of thing. think... vast majority of people in boardrooms, you they don't have to use a calculator, let alone understand how an LLM works. So probably entirely the wrong people are making the decisions about where this stuff should be used. You've got some like boardrooms taking a briefing from some big consultancy like, know, Bain or McKenzie or one of those. McKenzie, sorry, McKenzie is the investment fee. and McKinsey or whoever say, you've got to adopt AI, everybody's doing AI, you'll be left behind if you don't do AI. And they're like, OK, we'll do AI then, rather than try to actually understand what it's going to work for and what it isn't. So I think there's a great deal of money being spent on completely useless projects. And I think that's a shame because it's going to set, when a lot of this stuff come, when a lot of these chickens come home to roost, it's going to set back the adoption. by quite a bit I would expect. And then, you know, it'll be very easy to see in future, you know, there'll be big companies that could be really benefiting from it. I'm going to say really, I mean, you maybe it's a 10 % improvement in efficiency in a particular area. Who are, oh no, we got burned on that, we're not going to do that. And yeah, I don't think that's good. And it's a bit of a reflection of there aren't enough people that actually understand technology at the detailed level. sitting in the boardrooms of large companies with a lot of money to spend and some difficult decisions to make. think that's been the case for a long time. And that doesn't necessarily apply to tech companies, of course. I think it's telling that Amazon pulled back pretty rapidly because I think they do have people understand, still do have people understand technology at the very most senior level. But the big tech companies are unusual. They're also unusually profitable. some of those very big companies that aren't in tech and don't have tech on their senior leadership might want to think about what is it that these companies are doing that we're not doing. That's obviously a difficult decision when your company is run entirely by people with MBAs from Western or wherever. To do that is a valuable skill set, but it's not the only skill set.
Shubhendu: Mm. So if you Yeah, if you have to put a number, Andrew, like in terms of the percentage of how much of these big companies who are working on on AI or AI initiatives and how much of that is actually just theater, like in terms of actual spend actual hour. Just yes.
Andrew Haisley: Yeah, mean probably most of it. I mean, I remember the, you know, you go back to sort of the end of the 2010s, a lot of big Canadian companies decided that they needed to be more innovative, which is a fair point they do. And then they decided in typical sort of administrative fashion, the way to be more innovative was to create a department for innovation. Well, I can't think of anything less innovative than that myself. a lot of them created, probably still got these things, they created these innovation groups. Innovation theatre, I heard it described as. I think that's slightly unfair because some of the groups were given enough freedom to actually do things and did produce some interesting stuff. But the point still stands. think right now we've got an awful lot of AI theatre going on. much less useful AI. And what you've seen is a few companies that have taken the plunge into real AI have suffered, in some cases, have suffered quite badly. Was it Klarna? I think I'm thinking, I think it's the company I thinking of. Apologies to Klarna if it's not. They've adopted AI very heavily for their customer service agents and had a very bad result and had to pull back from that. And I think I'm hoping that some of the more sensible companies are looking at those experiences and saying, yeah, OK, maybe we shouldn't do that. I mean, it's a good example, actually. So I like YouTube. I think it's a great platform for learning useful things. But just really over the last year or so, it's started to fill up with AI slot. So somebody will use. and now I'll end to generate a script and then I'll use a different piece of generative AI to turn that into speech for a voiceover and then I'll go and get a bunch of video clips, some of generated, some of them just lifted from somewhere else, put them all together with this thing underneath and you click on the link and it looks like it might be a really interesting topic and then you listen to like a couple of minutes and just realize it's yet more AI slop. But what I've noticed
Shubhendu: Okay. Yeah.
Andrew Haisley: And then of course you ignore, I ever want to see this channel again. I think people are very good at realizing when they're being fed AI generated garbage. And I think that same thing applies to the output of big corporations. You you call up user support and you get an LLM that doesn't understand what he's talking about properly. You very rapidly get turned off using that company again. It's like, well, what am going to do if there's a problem with the service? I'm talking to this LLM here. I have no idea what he's doing. And yeah, that's a very good way of losing customers very quickly. I think most large companies are probably realizing that, yeah, that's going to be a problem. It doesn't mean to say there isn't a use for an LLM in customer support prices somewhere, but it has to be, what's the right word for it? A very thin veneer. on top of the real customer support. It might be that you've just got a simple question and if you train the thing correctly, it can answer you. But most of the time these days when somebody calls user support, it's because they've got a real problem and he's fixing. They didn't call up to ask, have you got any good deals at the moment? They called up because their internet's down and they need it fixed and the other end ain't going to do that for them. anyway, so yeah, to your point, yes, I think is the answer to that.
Shubhendu: Now I mean we discussed a lot of like what are the drawbacks, etcetera, what's happening in this industry, but it's I think you will ag agree in principle that AI is here. There it's a technology that's gonna fundamentally change how we work, be it whatever professional or you.
Andrew Haisley: Disagree. Well, the current AI is not meant to fundamentally change how we work. It's just another thing that will, when used correctly, make our lives a bit easier and a bit more productive. I think definitely it can do that. It's not going to fundamentally change it. Sorry, gone.
Shubhendu: Yeah, but still you so my question to you Yeah Yeah, my question to you is more from the listeners who because I I I have my newsletter and I I get like messages directly for people like who are feeling left behind, who are feeling that okay and again there's a lot of negative marketing around different companies as well about AI is gonna take away your job, etcetera. So people are actually generally worried and concerned.
Andrew Haisley: Mm. Yeah.
Shubhendu: And these people reach out to me and and ask me, which is I'm gonna forward you that question is if like where if they have been feeling left behind and they have not been like the person who have been trying trying Chat GPT or Claude from the get-go. If we're talking about the day today, what would you advise them? Where should they start? What should they do in order to kind of prepare themselves for what's coming? I know you said that today AI is not that like at that level that it's gonna disrupt.
Andrew Haisley: Hmm. you
Shubhendu: But still like to get started to c catch up, what would your like direction be for these viewers?
Andrew Haisley: I mean, I think it depends on what their job is. I think if we go to the, a technology person, and this could probably apply to everybody, developers, system administrators, support people, customer service, you name it, anybody that's doing sort of broadly tech job. I mean, the obvious place to start is just like. try to solve a problem using ChatGPT or Claude or whatever you have access to in your workplace, see if it helps. I mean, that would be my first thing if you're trying to solve a tech problem. The one actually quite strong piece of advice I would give is for any tech person that's got to a document, because most tech people hate writing. Right, so they hate writing human language. They're okay with like computer language. They hate writing human language. Well, actually that's an area where I guarantee that an LLM will help you. It will take a load off your mind. and write some bullet points or whatever the hell it is you're trying to say. Put it into an LLM and tell it to turn that into English text or French text or whatever it happens to be. And it will spit something out that's okay. It won't be beautiful. You're not going to come across as the next great... writer in literature, but it will be coherent. It will probably say exactly what you wanted to say. And it will actually quite importantly, it will say it in a way that most management types understand. So I would definitely start there as a techie, you know, get it, help make it help me write, write human language when I need it to, and also help get it to help me answer technical questions. Cause it will do that quicker than messing around with lots of internet search. But again, predicated on you knowing what the output should probably be approximately. I think In other jobs, it's a much harder question because, you know, if I'm an accountant, for example, can I use AI reliably and safely? Difficult because, you you've got this whole audit trail question. You know, I think, again, where you've got to write some text, it might be quite helpful. know accountants are a bit like programmers and they tend not to like writing very much. So yeah, again, that's a use case where it would probably be quite helpful. But I wouldn't, for example, want to copy and paste a balance sheet into an LLM, say, give me an analysis of this, and then rely on that answer without having gone through that whole process myself. I think that could lead to all sorts of difficulties. So it's a bit hard. I think across all jobs I can think of, using an LLM. to help you craft your written responses to things is not a bad idea. But only if you're somebody that doesn't like writing. If you like writing and you're quick at it, you'll probably do just as good a job. You'll do a better job and you'll do it probably just as quickly, ultimately, once you've been through an edit cycle. I think people are, too many people when they look at AI, because of that, it's so advanced, it seems like magic, think that... well, know, maybe they wouldn't think about it like this, but it would be, the Turing test has been passed. I'm dealing with a real thing here. Real. I'm dealing with something, an AGI and I'm dealing with something that's a person, but it's not. I mean, you can't say to an AI, hey, do my job. You know, people might think that, you know, you can do that with Claude Bort or whatever. What's it called? It's not called that anymore, it? Pardon? Yeah, that.
Shubhendu: Yeah. Okay. Full screw.
Andrew Haisley: People might think that that can do it for them. I'm sure there's some equivalents out there. Can't. You know, I mean, can make like really, really mess things up for them, but it can't take over and behave like a human would do in those situations. I don't, I think people are being hoodwinked over what this stuff can do. And it's not surprising when you have billions and billions of dollars being spent to
Shubhendu: Yeah.
Andrew Haisley: on marketing and advertising to convince them that this is the case. I think you've got to get hands-on in detail with the output of one of these things to really get a feel for what they're good at and what they currently aren't very good at. And I personally think we're hitting some, we're pretty much at the limit of what an LLM can do. I don't think...
Shubhendu: No.
Andrew Haisley: You know, it's the, when will we have AGI? another 10 years. I think we're in that stage now, right? You when we're going to have nuclear fusion, another 10 years, right? I mean, I think we'll have nuclear fusion before we have AGI, by the way, but that's a different conversation.
Shubhendu: So the honest message if I have to like synthesize what what you just said is we should learn the not learn the tool, but rather than what are the edges of the tool based on your job profile, what other things then the tool cannot do, you should know that. the limitation of that you can
Andrew Haisley: Yeah. And if you're a programmer, don't fall into the trap of using Claude for everything and getting other people to review out the problems. I this is a pattern I've heard from a few big tech companies actually, where developer will, particularly at the very junior end, having not really learned how everything fits together, will get Claude to do that piece of work for them. It will probably produce something which kind of works. And then they'll submit that as a, you you commit to whatever, you know, version management system they're using and ask somebody else to review it and the reviewer will look at it, the reviewer being a much more experienced person, they'll see just an enormous amount of code that doesn't necessarily do the job very well and then they'll go through this loop of essentially giving the person feedback, they'll put that feedback into interclaw and have Claude update its output and you go through this loop. I'm hearing that that's actually quite a common anti-pattern in larger environments and just My advice would be just do not fall into that trap. Because you'll never learn anything there. Well, ultimately, you'll probably get fired as well. It's very important that you understand what the output should look like. Then once you do that, and maybe you say to Claude, do this, you can see what his output looked like and think, it's not quite right. Give it updated instructions. It does a better job than you hand-aided the result. And possibly that took you less time than doing it yourself. I mean, particularly for...
Shubhendu: Yeah.
Andrew Haisley: things like user interface code that might work quite well because you tend to have to write tons of boilerplate and it's not concise. But anything where it's like you're dealing with a complex, highly optimized code base, using an LLN to make major changes to that is a recipe for disaster. And if you do that as particularly as a junior developer, you're going to damage your career quite badly, would be my expectation.
Shubhendu: Yeah, I I think I'll add one one thing there. a little different than what you said, but like those who are wondering where to start, what to do, the people who will win will not be the people who trust AI the most. Actually, they will be the people who know when not to. And that's very important to know the difference. There's a fine line of relying on the tool, which again using all these tools to kind of augment yourself is good, but then should know the limits of it and you should know exactly again adding to what Andrew just said that you should know exactly when not to what the what are the guidelines you should be putting in your work to kind of make sure you do not fall in that trap. But thank you so much for that Andrew. I'm sure the listeners, the viewers will appreciate that advice. Now we we do one thing every every episode, Andrew, and it this is like you asking a question for the next guest for our next episode.
Andrew Haisley: Yeah. Thank you. Yes.
Shubhendu: that next question is Michael McKernan. I think you guys have worked together in the past. so what do you want to ask him around AI?
Andrew Haisley: Yeah, we have. We have? Yes. So Michael's a very, what's the right word for it? broad-based guy, to put He knows a lot of things in lot of different domains. He's not your typical techie. My experience of working with him, which is good, by the way. I highly recommend knowing about lots of things that aren't technology as a techie. So I'm very, and something I've sort of wrestled with, particularly with the music side of things. I'm interested in his opinion of the uses of AI. in a business, professional, maybe scientific environment where possibly artistic quality isn't the number one requirement, right? Versus the use of AI in a creative environment. And, you know, my examples would be there's a lot of AI-generated music. And some of it's been quite successful. But to me, it all sounds... kind of soulless, you know, it doesn't have that sound of something produced by a person, you know, it doesn't have that, what's the word for it, torture in a spirit in it or however you want to put it. So I'm very interested in, like I said, Michael's a very broad-based guy with a lot of different interests. I'm interested in his opinion on the story, you know, where that line should be, you know, what should be. Where should we draw the line and say, yeah, AI's just don't generate good creative stuff? Where should we stop? Or maybe not where should we stop, but where does it stop? Where can people really start to tell the difference? That's my question. It's a bit philosophical, but I think you'll like that.
Shubhendu: Okay. I I think that that's a great question and I am sure it's gonna make him ponder and think a little bit. So Andrew, again, thank you so much for your time today. This is generally one of the most interesting and in like enriching conversations I've had recently.
Andrew Haisley: It's fun, I like to get to talk nonsense for a while.
Shubhendu: But I'm sure this is not the the last time we're speaking on this show, Andrew. So again, thank you again for your time. we'll be back next Monday with a new speaker for you. So stay tuned. If you like, please do do not forget to comment, subscribe. And if you have any thoughts, any feedback, anything you wanna listen to, any topics you wanna cover in in my next episode, please do not hesitate to write it down in your comments below. Thank you everyone and thank you, Andrew.
Andrew Haisley: For whom?
Shubhendu: Everyone is talking about AI. Almost nobody is telling the truth. My guest today has spent more than 30 years building and leading software teams, from startups to Amazon itself. He's worked at Royal Bank of Canada, also at Wayfair. He's worked in three countries, Canada, US, and UK, across three different many different industries. And he has now retired to spend more time on the tech he actually finds interesting. He's not selling anything, and this is exactly why I wanted him here. Andrew Hazley, welcome to the integration layer.
Andrew Haisley: Hello, nice seeing you again, Sue, Chip. Been a while.
Shubhendu: It's been a while. so thank you again for taking time today, Andrew. I want to start s with something you said to me a few days ago that really stuck with me. you have a simple rule about how to assess if AI works for a certain use case and not. Can you walk me through that?
Andrew Haisley: you But it's not so much assess. I mean, I think to actually assess the use case, you have to do it and take a view on the results. But I think the rule that I apply to figure out beforehand a priority, is this thing have a decent chance of success or not, is the thing I described to you. It's the relative size of the input to the output. If you ask an AI to take a large amount of input and summarize it, Generally speaking, will do an extremely good job. And I think I first used this at Wayfair since you mentioned that, three years back now, or whenever it was, it's been a while. Early days of chat GPT, feed in a long document on a particular topic and ask it to produce an executive summary. does a very good job. And it's good at figuring out, or say figuring out, I don't want to say that because that's not what it does. is good at identifying those things which are most relevant. So very good in that use case. And I've used it for that sort of thing a fair bit since. It's also good where the size of the input is kind of the same size as the output. So an example of that would be, I act as secretary to a couple of small organizations and I make notes during meetings, know, scribble things down, type bullet points out. And it's very good at taking, you know, five or six bullet points on a topic and turning it into text describing the same thing. Surprisingly good actually, because often I'll put a bunch of stuff in without a lot of context and just say, turn this into a paragraph of text, for me, will you? And it actually does a good job of it. So very good for those kinds of cases. Also good for very small, narrowly defined tech cases. Now, some of these, the actual output might be a bit bigger because it's code rather than text and code. There's a lot less information rich than the kind of problem that generate. So, random example, I was playing around with bit of very old software, actually, the other day, some software back from the 90s, C code. X Windows, Unix stuff. And I decided, there were a lot of reasons behind this, but I was trying to port this particular piece of software to use X FreeType, which is the modern, I say modern, but like invented in the late 90s font technology that you see everywhere, but don't know, don't, you don't necessarily know that it exists. And ChatGPT was extremely helpful about, well, give me a bit of sample code that uses this function to do this. Very good at that. Definitely saved me a load of time. So those are the kind of cases where it works really, really well. But then when you go to the case where you give it a small amount of input and say, me a lot of output, I've never seen good results from that. I mean, let me give you one concrete recent example. So I play the French horn not very well, but I play it. And a particular challenge of that instrument is just playing the right note. It's not like, say, a piano where the challenge is playing 50 notes at once very quickly or various things like that. I'm not a piano player, so I don't want to sound flippant about that. But the right note is evident. You can press the key that has the right note and you get the right note. The French horn is not really like that. large part of the effort of being a good porn player is being able to just get the right note out when you need to and there's many reasons why that's difficult on that instrument. So it occurred to me it would be nice to have an app that would randomly display a note, ask me to play it, sample what I played, tell me whether I was accurate or not. Not conceptually a very difficult thing to do. There's plenty of tuner apps out there you can already get that will tell you what note you're playing and simply generating a random note and then comparing the two would seem very straightforward. So I put a short spec for that. I actually did it twice to do the Claude as well, Claude and cursor. So I put a short spec for that app into the AI, Claude command line and cursor equivalent. with that information saying, create me a Swift app to do that. I thought I'll do an iOS app, because I'm an Apple user. And well, what came out was garbage. mean, it was able to produce an app that started. That's pretty impressive. I mean, that in itself is useful, given I've never written a Swift app. That would definitely save me time if I was starting from scratch. But in terms of the app itself, there were all sorts of things wrong with it. I had no idea what it thought it was trying to do when it came to figuring out the pitch of the note, for example. Didn't use any technique that I'm aware of that would make that work. And also actually now I remember it, it couldn't really draw the notes properly, it put the lines on the wrong side. It was all over the place. And I did have a look through the code that was generated and it was dense and pretty obtuse. That doesn't surprise me. If you think about if you were to take, a very junior software developer who didn't know anything about music, had no domain knowledge, had never really written an app before, or least never written anything themselves from scratch, and gave them that description, they might welcome up with something just as bad. But I think a lot of people get fooled into thinking because the AI can spit that stuff out quickly. it's going to produce something that works correctly. And I just don't think that's the case most of the time. It might look like it works sometimes. I for more commonplace types of application, it might produce something that kind of seems to do the job. Reality is, that most of the time it's producing very, very poor quality code. Now, that's not to say it's not useful. Given enough time and enough description, it probably could have produced an app to do what I wanted. but I'm pretty sure I could have written it quicker myself with much less frustration and probably better quality code. So I hope better quality code. So I think, you know, with those three distinct use cases. The common factor is the size of the input versus the size of the output. And I think that's a pretty decent rule of thumb as to whether this thing's likely going to work or not.
Shubhendu: So if I summarize correctly, if understand this right, AI is a brilliant compression engine as per you and a terrible invention engine. It's interesting because after we discussed about this, I was actually going through some notes of mine from PR. So in 2025, there's a research group which ran a proper randomized control trial with 16 experienced developers on real tasks.
Andrew Haisley: Hmm.
Shubhendu: And the developers predicted that AI would make them faster and roughly twenty plus percent afterward. They still believed that it had made them faster, but when the group measured it again, the actual time, they were actually nineteen percent slower than with AI.
Andrew Haisley: Thank Was that the Carnegie Mellon study?
Shubhendu: no, it's group called E T R and this was published in July. So they were actually yeah, there was actually nineteen percent slower. So the feeling and the reality point in opposite directions. Why do you think that is? Like why experienced people feel faster than us?
Andrew Haisley: I'm pretty sure. Yeah. Well, it lulls you into this false sense of security, doesn't it? You say, do X, and it produces something that's kind of like X. And you didn't have to spend all afternoon thinking about how to do it yourself. It's like, you just did it in half an hour, and there you were. You had something that kind of looked like X.
Shubhendu: I mean
Andrew Haisley: I think you incorrectly interpolate that or predict from that that it's going to do similar things for the whole of the lifecycle of whatever problem it is you're trying to solve. So you might, I think something I said previously when we were talking, if you take the lifecycle of a software problem, product, whatever it is, you've got this much stuff and things happen over the lifecycle, what the AI is very good at is moving things from one part of the lifecycle to the other. It doesn't necessarily shorten it. It's that sort of adrenaline rush of getting an absolute pile of stuff done really, really quickly, followed by the kind of like whatever the opposite of an adrenaline rush, the letdown and realizing that what it did wasn't very good and you now got to go all the way back through it and rework it. So that study result does not surprise me at all. And I think the interesting thing.
Shubhendu: Yeah.
Andrew Haisley: Yeah, the interesting, the thing that people should think about carefully. And I'm sure this knowledge will develop with time is learning to use the AI to speed you up reliably. And it's back to, all the way back to Fred Brooks, there is no silver bullet, whatever it was called. Well, there's still no silver bullet. is no, as I think I'm going to paraphrase Brooks here.
Shubhendu: Yeah.
Andrew Haisley: You might want look it up to make sure I got it right later, but it's something like there's no innovation in technology or management process or technique that will produce more than an order of magnitude improvement in productivity within a decade. And that's actually very optimistic that there may actually be techniques which will double your product. And he was talking double, not base 10. double your productivity in a decade, that would actually be really, really impressive. And of course, he was starting back in the days where they moved from assembler to, know, alcohol or some cobalt or some high level language. And yeah, actually it probably did double people's productivity because, you know, I've written assembler, it's hard work, it takes a lot of time. Much easier to write it in a high level language. So, I think that still applies. I've not seen anything about AI. or what we call AI. Actually, let's talk about that in a minute. I've not seen anything about AI that would make me believe that we've somehow broken, have a solution to that rule. It just doesn't jump out at me if it's there. And I think the problem is that people are confusing AI, which is LLMs, effectively. There are other things, but what we're really talking about here is LLM. LLMs, people are confusing that with AGI, know, artificial general intelligence. It's not, you know, it's a very, very clever inference engine. Some amazing stuff in there and some very interesting kind of rule sets written on top of that. You take something like Claude Coates, very impressive what it can do. But we still have not solved the symbolic reasoning problem, which was where AI was sitting back in the 80s when I
Shubhendu: Yeah.
Andrew Haisley: did my undergraduate degree. That problem hasn't been solved. It's not, know, the LLMs have not come along and sort of produced a general theorem solver that can under all circumstances understand that it's done something that makes sense or not. It's still using the same underlying LLM techniques. Now, there may be some progress from here. might be the whole, everything that's been built within the last few years. triggers some new direction of research, which leads to the other problem being solved. And then we have AGI, and then there's a whole different conversation, but we're not there, not close.
Shubhendu: Yeah, I agree. And I build these systems for a living and at Cubitron what we have what we've noticed is you mentioned about Cloud Code, funny enough that we use it a lot. What we'll notice is AI is helping like either remove or or decrease the boring stuff, but still the thinking, the debugging, n the saying what is right or what looks right, I think that's doesn't go away. That's very important of and that's something like e despite of
Andrew Haisley: Yeah.
Shubhendu: Giving these tools to the development team at Kilitron, for example, I still do see various amounts of different kind of output from different team members. And that reflects their seniority, their expertise, their experience, etc. So 100% agree on what you said. Now, that is the overhyped side. Let's talk about something which you generally think is huge, which is nobody's almost something nobody's talking about. you told me you told me something
Andrew Haisley: Yeah.
Shubhendu: ab about like broadly solving the semantic search problem. we talked about Google, you talked about the feature rank system.
Andrew Haisley: yeah, yeah, yes. No, absolutely, it's interesting that this doesn't seem to be much of a topic of conversation, but if you go back to the 90s and the invention of the search engine, you had the early ones like Yahoo and AltaVista and a load of others that I've probably forgotten, which were essentially keyword search. I mean, they attempted to be clever about it, but it was really pulling all these websites, store some index of keywords and let you search those. And they were fairly terrible. I mean, they didn't, if you put a few words in, was highly unlikely that any of those search engines would come up with what you were really looking for. There were a couple of exceptions to that, which took much bigger blocks of text, but they were never particularly widely adopted. And then Google came along with the PageRank algorithm, which made no real attempt to understand what was actually on the website. just said, essentially all the people that thought this particular set of words meant something linked to this particular page here, therefore it must be a pretty good place to go look for those words. So it was a kind of heuristic based thing. A little bit like, you know. LLMs with a statistical approach. And that took off. But the huge topic of conversation before Google came along was this concept of semantic search. So the idea was that, well, the problem to solve is you should be able to just type some kind of sensible question in human language into a search engine. should figure out what you mean by the question and then go and find pages that answer that question with some sensible response. And of course, that's just... when you think about it, that's really just one of the old AI unsolvable problems. I've got books on those shelves over there about natural language processing back from the 90s. And they're very interesting, very technical, they don't actually give you any particularly useful solutions to that problem. But that was the big topic. And then Google came along and everybody forgot about it because PageRank worked well enough. But finally, with LLMs, we do actually have something that provides something that looks like a solution to semantic search. I'm not going to say it understands what you're asking or anything, but it produces a response that generally, as long as you know kind what you're looking for, looks like it knows what you meant when you asked the question. So I can go to chat.gpt, for example, because I actually just did a while ago, and ask it, is it reasonable? when transcribing a piece of orchestral music for concert band to give the viola part to the alto sax in the concert band? Right, very open question. If you try Google search for that, you don't get anything useful at all. You get various things that might be tangentially useful. Give that to ChatGPT though, and it gives a pretty sensible answer along with links to pages discussing that exact problem that don't show up in regular search using that search query. That I think is incredibly useful. But back to what I said earlier, it really depends on you having a very good idea of what the right answer should look like. Otherwise you could make all sorts of stupid mistakes.
Shubhendu: Yeah. Yeah, and I think for the viewers here or for the listeners here, you you sent me you t told me the example of the green card. I think that was very perfect it to understand.
Andrew Haisley: That's a really good one, yeah. We can talk about that as well. mean, that's just a random example. Well, actually, this is a good example because it shows where the AI got the right answer, whereas search produced very clearly the wrong answer. I've got examples the other way around as well. So I'm sure many people have heard of the US Green Card Lottery, also known as the Diversity Lottery. And it's an American program that's been going forever. huge source of scammers, I suspect as well, where you can apply for a green card and you get picked on lottery basis. So there's some basic entry requirements. And I was talking to somebody the other day who was just joking about thinking about moving to the US, but he was too old. And I said, you got the green card lottery. You could do that. And then he said, well, is there an age limit? I was like, okay, that's interesting. don't know, I no idea. We'll go and have a look. Do a Google search for it and you you hit all sorts of AI slot websites which tell you there's an age limit of 26, 27, depending on which site you look at. And I thought that sounded a bit odd. So I went and asked, chatgbt, and it gave me the right answer, which is there is no age limit. And it gave a link to the USCIS or whatever it's called, the US immigration. government immigration website which laid out the criteria. So I that was a really good example. And I think the reason that the, and I think this is kind of ironic, the reason that the LLM was able to give me the right answer is, and it's ironic because all of those websites, slot websites are generated using LLMs. I think the inputs used to train the LLMs, they're much That's the right word for it. They're much better at filtering out swap websites and the general search index algorithms are. So they train their LLM on real websites and they've got to the point of being able to figure out what's an authoritative source. Whereas what you hit with a standard search is an index of just about everything. They don't seem to be making any great effort to filter that stuff out. might be somewhat deliberate because they want to drive you to use their LLM as opposed to the search stuff or more likely than not it's because all of the code used to do the indexing for the search side is in a different silo from the people that do the data gathering for the LLM side and then figure out how align them yet. But that's an example in that direction. I can give you good example in the other direction though where the LLMs are just plain wrong. Would you like to hear about that?
Shubhendu: Mm. yeah, actually just before that, because that's very relevant to what I'm about to share, after we discussed this, I actually looked up some reports and this is just an estimate, but like it's still mind-boggling to understand that as per the last year, so 2025, the number of Google searches that led to a single or any website click has fallen by more than 60%. So like When we search and we used to go and look for links and and go to various websites, now people are looking at the AI AI overview and then just not going anywhere. now like we have said Google search was that the when Chat GPT was launched, but now the answer in certain cases are is coming on even in Google without any source. Or in some cases without a source. What do you think we we we lost in that trade and like
Andrew Haisley: Yeah. Yeah. Awesome.
Shubhendu: Yeah. Tell me more about putting that link.
Andrew Haisley: I think there's two things we are losing. I think to be fair to the LLM people, because they are now in many cases embedding links in their responses so you can see where they got the answer from. To be fair, I think they do understand this problem. But I think there's two things. We're losing, you could say that the internet. was a downgrade from using a reference library prior to that, so was all along the same path. But people are losing the ability, maybe not the ability, but the willingness to research and answer properly. That green card question, for example, wouldn't be, actually that's a bad example because the LLM got it right and didn't, but where the LLM just gives you an answer and it kind of looks fairly okay. and you don't go and click on the links and try to understand how it generated that answer. I mean, you're not using your brain there in a way that you probably should. Even if the answer's right, actually, you should probably go and look at the source material to understand it. So I think we've gone from a situation where if you wanted to find out about something that wasn't just easily available, you go to a reference library and start with the index and maybe ask the librarian and... find a book and read a bit of it and have a look at the references at the back of the book and go from there. We've gone from you ask a question, you get an answer on what might be a very obscure topic and you just accept the answer. So you've lost two things. You've lost probably the truth because there's a good chance the answer is wrong when it's something obscure and you've lost the ability to do that research yourself. And I think that's bad. I don't think that, you know, I'm constraining this discussion to technology. I don't think that bodes well for the future quality of software. think when you have a development team that just takes the answer it's given from the LLM and uses it without trying to understand, well, why did he give me that answer rather than this other answer? That's not going to make your software better. I'm very certain about that.
Shubhendu: Yeah. Yeah. And yeah, some like s same thought on that because earlier we used to search and when they used to see okay, this is some coming from a published white paper or published paper from some reputed PhD graduate or some someone like that. Or this is coming from Tommy on Reddit, who is ranting about something. And we used to make a judgment call that time. And it was inherent whether we were searching about something on on Stack Overflow or we were searching something about any kind of
Andrew Haisley: Yes. Thanks
Shubhendu: like tourist places or restaurants. But I think now the more despite that you just said that all these tools are giving the sources, how many can you expect, right? So eventually or some of the percentage will fall through gaps. We'll not check, okay, I'm looking for an answer. I need it now. I'll just s ask chat GPT or or Claude and I'll just believe what I see in the first place. Very few people at this point, because
Andrew Haisley: Hmm.
Shubhendu: Earlier it used to be okay, I need to go and check those links based on the age rank. So now people are like, like I I can't imagine in in the way tools are designed today, that how can we actually code those things? And how many quotes are we gonna bring? Okay, this is the sources are X, Y, Z, but how many of them are you gonna show? Because that's also gonna bring another problem and that's for a different time for our discussion is the tokens. The more of the output the more of these so
Andrew Haisley: Mm.
Shubhendu: information is given the more token consumption is gonna come back to. Which anyways, but this takes us to something that we you actually have been testing for years, whether these tell you the truth. So get to happen. Yeah.
Andrew Haisley: Yeah. Yeah, so let me talk about this. a controversial topic and I absolutely don't want to discuss the topic itself because it's been done to death. But I started using this as a test because it fits very nicely the pattern of something that was widely reported or widely written about in one way, but which... later on in a much more narrow, authoritative way was shown to be false. And, you know, if you're an expert in any particular technical area, you can go and find examples of this, but this one works because I think very many people, particularly in Britain, are familiar with it. And undoubtedly, if this video is going to be on YouTube, you're going to get some comments underneath that say, say this isn't the case, by the way, which would be quite entertaining. But the Brexit referendum, yeah, already we're getting negative comments here. There was a widely reported situation involving a company called Cambridge Analytica. And the reason I got interested in this was I listened to a very good Michael Lewis podcast.
Shubhendu: Yeah.
Andrew Haisley: on the subject Cambridge Analytica. Michael Lewis is a very well-known author. wrote Liar's Poker, The Big Short, Moneyball, a whole bunch of other books. Very, very good writer. Used to work in financial services. Writes from a position of knowledge, but he's also very non-partisan. He just looks at stuff and says, this is what I think is going on here. So he did a very interesting podcast on Cambridge Analytica. Now this was a company that essentially their cell, well they did various things, part of their cell was directed advertising on Facebook. And they got into trouble because they used data taken. from Facebook that breached various privacy rules and things in various places. They got into trouble, they got fined, I don't think they exist anymore. the Michael Lewis thing was interesting though because he interviewed somebody who'd worked for them, think right at the beginning of their existence, maybe a bit later. But he talked about how what they were selling just didn't work, or least didn't work to any meaningful degree. Maybe it had a tiny... tiny impact. well, you know, it just couldn't possibly work for all these reasons. And he just found the whole scandal a bit silly because, you know, they were selling a product that couldn't really do anything. the story at the time with the Brexit referendum was that Cambridge Analytica were involved with the leave side, which is why it's controversial, right? And their technology had somehow influenced the result of the referendum.
Shubhendu: Mm.
Andrew Haisley: which given, I think it was a million person margin would have been quite impressive. Facebook posts persuading a million people to vote in, well, half a million I suppose, vote in a different direction seemed implausible, but it didn't matter. There were a lot of stories in the press about this and reputable press as well. We're not talking Reddit here. it would have been on Reddit, but reputable press, BBC covered it, Guardian covered it. probably the New York Times, I haven't looked, but know, a bunch of big, big trusted media sites covered it and explained how Cambridge Analytica were involved and they may have had some impact on the results and it was a scandal and it should be investigated, et cetera, et And in fact, I think one journalist might even have won kind of amusingly an Orwell Award for writing about this. Just kind of entertaining. But anyway, it passed me all by at the time until I heard this
Shubhendu: Yeah. Yeah.
Andrew Haisley: this podcast and then I thought I'll go and look into this into some detail and I discovered was that The British Information Commissioner had got involved a bit later on and carried out a full legal inquiry, know, the kind of inquiry that has the power to force people to provide documents and requires people to turn up and testify and so on. So a proper, you know, not just a sort of superficial thing, you know, the real judicial level inquiry with enforcement. capabilities and stuff. And she looked into it in a lot of detail, got all the evidence. the summary was, and you can go find this if you're interested, the Cambridge Analytica had maybe one or two initial discussions with the UK Independence Party, who said they weren't interested in hiring them. And that was it. They had no actual involvement in the referendum at all. They just weren't involved. Full stop. Now, the interesting thing is that was not that widely reported. I mean, there might have been the odd follower particle, but there were orders of magnitude more places saying that, they were involved and they had an impact than this lone voice from the authoritative source saying, no, actually not. I think it's because the world had moved on a bit by the time that came out, so it wasn't really an interesting topic anymore.
Shubhendu: But still, it's it's very sensitive because the person in question for the viewers, the person in question here is the UK Information Commissioner Elizabeth Denham. So in October 2020, the conclusion that she brought was Cambridge Analytica was not involved in the Brexit referendum beyond some initial inquiries. And this is coming from the UK, the her report to the parliament. And the micro-targeting that they claim to do, investigators found that the effect was close to zero. Now, for three years. The AI would confidently tell you that a Cambridge Analytica swung the vote. You had to push before
Andrew Haisley: Yeah, but if you can quite say it like that. if you went to, I tried this right back when CHAP GPT came out. that's interesting. Because that's just a really good example of something where all the data says one thing except for the critical piece, which says the other, right? And so I asked CHAP GPT, Gemini, and Claude.
Shubhendu: Yeah.
Andrew Haisley: around about the same time, well, what was Cambridge Analytica's involvement in the Brexit referendum? And they gave me this great long thing about how they were involved and all this stuff and that they may have swung the vote and blah, blah, blah. They gave me the, I guess, the orthodoxy on the subject. And then when I said, well, hang about, the information commissioner said this, and that's an authoritative source. Even when I said that to them, they pushed back and said, well, yes, but. And then I said, well, no, this is the authoritative source, what you're talking about. And eventually after several goes round, They were like, yeah, you're right. That's what happened. And I've been trying that again every now and again, and they've got better. They're not perfect. And it's a bit random, of course, because you've got some randomness in the responses here. But they are heading towards the point where they do give the correct answer. They're not there yet, but it's got better. And I think this is, I'm assuming, is because they've got better at awaiting truly authoritative sources versus scuffle butt basically. But it's interesting, you you think well there's going to be a lot of things like that. That one I just happen to know about. But I asked, you know, I asked the AI all sorts of things. I don't know how many times I'm being told basically the Reddit consensus answer versus the reality, you know.
Shubhendu: Yeah. Yeah. So if AI just keeps like echoing the loudest story now who's accountable for that? Is it like if this any AI, whether it's Gemnai or or Chat GP or Claude, if it tells a hundred million people, any confident wrong answer, right? So and that's the biggest concern which we have around.
Andrew Haisley: Yeah. What is the AI any different from a human? mean, there's plenty of people spreading this information out there. So I think you've got to answer that question before you can answer is the AI in some sense responsible for its output? mean, you know, at the extreme, you can take somebody like Alex Jones, for example, and, you know, eventually the legal system dealt with him in some way, I guess.
Shubhendu: That's a good point. Yeah, but
Andrew Haisley: I don't know what the answer to this is. I don't know that it's reasonable or indeed practical to hold an AI company liable for the perfection of the answers that its product spits out. you know, who's the... I mean, in many cases, who is going to be the arbiter of the truth? If it's a piece of maths or a of physics, there's, okay, definably a wrong answer. Well, even in those cases, there's not necessarily a... I guess if you go to chat GPT and ask it, which is right, Mond or dark matter? I mean, we don't know what the answer to that is, right? So don't know what it's going to say. So I don't know that in many cases there is... you know, universal universally agreed, improvable, true thing. That said, the stuff which is so obviously disinformation. mean, if you don't get into another another like difficult topic, but you know, there are there are clearly things around like vaccines where there's some very, very good research that shows certain things. And then there's some very loud opinions that those things are not true. If you ask the AI, I would really hope it would tend towards the published peer reviewed evidence rather than the loud voices. And if they allowed it to go more to those loud voices, you could argue that that is disinformation, has some pretty negative health impacts for people. I'm not sure there's... In fact, if I remember, while I'm thinking about that actually, I remember, wasn't that one, this seems to have died down a bit, but back in the nineties, there were a lot of...
Shubhendu: Yeah that
Andrew Haisley: miracle cures for cancer being pushed. you know, I'm sure if the LLMs had been around then they would have picked up that stuff and propagated. And it wasn't, know, this was a complimentary medicine where it's like, well, it wouldn't do any harm. It might make you feel a bit better. So fine, go for it. It was, no, all of that, all of that chemotherapy you've been told to take, that's going to kill you. should buy our, powdered muscle shells, I think was one of them. product and you know take that instead you know that's like seriously dangerous misinformation right. So I don't know I think there is an argument there but I think also governments trying to decide who is like telling the truth and who isn't is a pretty slippery slope too.
Shubhendu: Yeah, and that's what keeps me up at night sometimes because if if the AI is as honest as what it learned on the internet till that day, and on on the other hand, there let's like come to the boardroom where companies are betting billions of dollars in this exact this technology. So like let me read this out for you. Like yeah.
Andrew Haisley: I'm proud for the boat. Mmm. you
Shubhendu: In 2025, MIT's Nanda initiative studied 300 real enterprise AI deployments. 95% of the companies had seen no ROI, no measurable gains. And that is against an estimated $30 to $40 billion invested. The projects that work were narrow and specialized. The ones that tried to do everything stalled. And this is a research like a report coming from MIT straightforward.
Andrew Haisley: I'm not sure that's what asking. Yeah.
Shubhendu: 95% got nothing measurable. And you're talking billions here. And the spending is continuing to climb. Is that a technology problem or a story problem in your opinion?
Andrew Haisley: I don't, well, it's a typical problem of, you you've got a howling, you're looking for nails to hit, isn't it? You know, I saw the same thing, I mean, a much sort of lesser scale, you you could pick, say, some of those robotic process automation tools that were widely adopted by big companies. You know, I think most of those projects saw very little ROI. And certainly if you looked at the lifetime of the product, it would have been... very negative because they embedded all sorts of problems that were then going to be very difficult to fix later. But that doesn't mean to say there weren't any good deployments of RPAs. I bet there's some out there that were really good and work very, very, very well. I'm pretty sure the same would apply to AI if you just come in and say, we need to use AI everywhere and get going. People are going to try it and do all sorts of crazy stuff that doesn't really work. But if you say, I don't know. We've got this very constrained problem around interpreting this existing data set that gets updated regularly. And we've been doing this thing by hand forever, and it takes somebody a week to go through it. Can we train an AI to do that same thing? You might well get a really good payback. I'm thinking things that look through, collection of accounting data, for example, looking for kind of interesting patterns and stuff. If you're very constrained with the question you're asking and you've run through that process multiple times and calibrated the output versus the input using a human, I can see that that potentially would allow you to automate something in a way that you couldn't previously have automated and save yourself an awful lot of time in the future. mean, a good example would be You know, like I mentioned, I I mentioned this earlier about writing meeting minutes. That's clearly a repeatable thing that works. Saves me a lot of time. I'm very confident that over the life cycle of that project, assuming I live a few more years, I'll have saved myself hundreds of hours of time. I think that's brilliant. I don't see why that same thing shouldn't apply to certain categories of business problem. But very definitely the scatter gun we bought Claude go use it to solve everything approach is just not going to work. read recently that Amazon have backtracked on this quite a bit internally. think they had some kind of dashboard that would show who is using AI, how to what extent, and that was part of that person's sort of KPIs. But I they realized.
Shubhendu: Mm-hmm. Yeah.
Andrew Haisley: that then people were just burning credits to get themselves high up on this list and not necessarily producing anything useful with it. So they backtracked on that and they now just say quite sensibly, think use AI where you think it improves your productivity. And over time, you know, the, it's the right word for it, the results will be clear from them internally in some way the scale of Amazon, you know, they'll see certain employees are they're most productive and they're like being Amazon, they'll probably start measuring, you know, their other productivity metrics versus that person's AI use and come to some conclusion about how good AI is for them. I think same thing would have, I mean, that's something you can do as a huge company with hundreds of thousands of employees, a bit difficult if you're a 20 person company. But I think larger companies will probably move to things like that, know, measuring where it's generally used AI that any employee can pick up and do things with. They'll move to.
Shubhendu: Mm.
Andrew Haisley: correlating actually useful KPIs versus AI adoption and form their own opinion as to where it makes sense.
Shubhendu: Yeah. So you Andrew, based on your experience, you've sat on these boardrooms in these leadership like circles for a long time. What do you think is one of the biggest or most expensive thing that these companies who are l trying to implement or leverage AI, they're betting on, is simply false or unattainable based on what you know today?
Andrew Haisley: Thank I'm sorry. Well, they're betting on it being magic, aren't they? You know, what's that? That joke? Not joke, but that quote, you know, it's like, any sufficiently sophisticated technology will appear as magic to somebody that doesn't understand it. And I think, you know, I'm going to be direct here. I'm not looking for a job anymore so can say this sort of thing. think... vast majority of people in boardrooms, you they don't have to use a calculator, let alone understand how an LLM works. So probably entirely the wrong people are making the decisions about where this stuff should be used. You've got some like boardrooms taking a briefing from some big consultancy like, know, Bain or McKenzie or one of those. McKenzie, sorry, McKenzie is the investment fee. and McKinsey or whoever say, you've got to adopt AI, everybody's doing AI, you'll be left behind if you don't do AI. And they're like, OK, we'll do AI then, rather than try to actually understand what it's going to work for and what it isn't. So I think there's a great deal of money being spent on completely useless projects. And I think that's a shame because it's going to set, when a lot of this stuff come, when a lot of these chickens come home to roost, it's going to set back the adoption. by quite a bit I would expect. And then, you know, it'll be very easy to see in future, you know, there'll be big companies that could be really benefiting from it. I'm going to say really, I mean, you maybe it's a 10 % improvement in efficiency in a particular area. Who are, oh no, we got burned on that, we're not going to do that. And yeah, I don't think that's good. And it's a bit of a reflection of there aren't enough people that actually understand technology at the detailed level. sitting in the boardrooms of large companies with a lot of money to spend and some difficult decisions to make. think that's been the case for a long time. And that doesn't necessarily apply to tech companies, of course. I think it's telling that Amazon pulled back pretty rapidly because I think they do have people understand, still do have people understand technology at the very most senior level. But the big tech companies are unusual. They're also unusually profitable. some of those very big companies that aren't in tech and don't have tech on their senior leadership might want to think about what is it that these companies are doing that we're not doing. That's obviously a difficult decision when your company is run entirely by people with MBAs from Western or wherever. To do that is a valuable skill set, but it's not the only skill set.
Shubhendu: Mm. So if you Yeah, if you have to put a number, Andrew, like in terms of the percentage of how much of these big companies who are working on on AI or AI initiatives and how much of that is actually just theater, like in terms of actual spend actual hour. Just yes.
Andrew Haisley: Yeah, mean probably most of it. I mean, I remember the, you know, you go back to sort of the end of the 2010s, a lot of big Canadian companies decided that they needed to be more innovative, which is a fair point they do. And then they decided in typical sort of administrative fashion, the way to be more innovative was to create a department for innovation. Well, I can't think of anything less innovative than that myself. a lot of them created, probably still got these things, they created these innovation groups. Innovation theatre, I heard it described as. I think that's slightly unfair because some of the groups were given enough freedom to actually do things and did produce some interesting stuff. But the point still stands. think right now we've got an awful lot of AI theatre going on. much less useful AI. And what you've seen is a few companies that have taken the plunge into real AI have suffered, in some cases, have suffered quite badly. Was it Klarna? I think I'm thinking, I think it's the company I thinking of. Apologies to Klarna if it's not. They've adopted AI very heavily for their customer service agents and had a very bad result and had to pull back from that. And I think I'm hoping that some of the more sensible companies are looking at those experiences and saying, yeah, OK, maybe we shouldn't do that. I mean, it's a good example, actually. So I like YouTube. I think it's a great platform for learning useful things. But just really over the last year or so, it's started to fill up with AI slot. So somebody will use. and now I'll end to generate a script and then I'll use a different piece of generative AI to turn that into speech for a voiceover and then I'll go and get a bunch of video clips, some of generated, some of them just lifted from somewhere else, put them all together with this thing underneath and you click on the link and it looks like it might be a really interesting topic and then you listen to like a couple of minutes and just realize it's yet more AI slop. But what I've noticed
Shubhendu: Okay. Yeah.
Andrew Haisley: And then of course you ignore, I ever want to see this channel again. I think people are very good at realizing when they're being fed AI generated garbage. And I think that same thing applies to the output of big corporations. You you call up user support and you get an LLM that doesn't understand what he's talking about properly. You very rapidly get turned off using that company again. It's like, well, what am going to do if there's a problem with the service? I'm talking to this LLM here. I have no idea what he's doing. And yeah, that's a very good way of losing customers very quickly. I think most large companies are probably realizing that, yeah, that's going to be a problem. It doesn't mean to say there isn't a use for an LLM in customer support prices somewhere, but it has to be, what's the right word for it? A very thin veneer. on top of the real customer support. It might be that you've just got a simple question and if you train the thing correctly, it can answer you. But most of the time these days when somebody calls user support, it's because they've got a real problem and he's fixing. They didn't call up to ask, have you got any good deals at the moment? They called up because their internet's down and they need it fixed and the other end ain't going to do that for them. anyway, so yeah, to your point, yes, I think is the answer to that.
Shubhendu: Now I mean we discussed a lot of like what are the drawbacks, etcetera, what's happening in this industry, but it's I think you will ag agree in principle that AI is here. There it's a technology that's gonna fundamentally change how we work, be it whatever professional or you.
Andrew Haisley: Disagree. Well, the current AI is not meant to fundamentally change how we work. It's just another thing that will, when used correctly, make our lives a bit easier and a bit more productive. I think definitely it can do that. It's not going to fundamentally change it. Sorry, gone.
Shubhendu: Yeah, but still you so my question to you Yeah Yeah, my question to you is more from the listeners who because I I I have my newsletter and I I get like messages directly for people like who are feeling left behind, who are feeling that okay and again there's a lot of negative marketing around different companies as well about AI is gonna take away your job, etcetera. So people are actually generally worried and concerned.
Andrew Haisley: Mm. Yeah.
Shubhendu: And these people reach out to me and and ask me, which is I'm gonna forward you that question is if like where if they have been feeling left behind and they have not been like the person who have been trying trying Chat GPT or Claude from the get-go. If we're talking about the day today, what would you advise them? Where should they start? What should they do in order to kind of prepare themselves for what's coming? I know you said that today AI is not that like at that level that it's gonna disrupt.
Andrew Haisley: Hmm. you
Shubhendu: But still like to get started to c catch up, what would your like direction be for these viewers?
Andrew Haisley: I mean, I think it depends on what their job is. I think if we go to the, a technology person, and this could probably apply to everybody, developers, system administrators, support people, customer service, you name it, anybody that's doing sort of broadly tech job. I mean, the obvious place to start is just like. try to solve a problem using ChatGPT or Claude or whatever you have access to in your workplace, see if it helps. I mean, that would be my first thing if you're trying to solve a tech problem. The one actually quite strong piece of advice I would give is for any tech person that's got to a document, because most tech people hate writing. Right, so they hate writing human language. They're okay with like computer language. They hate writing human language. Well, actually that's an area where I guarantee that an LLM will help you. It will take a load off your mind. and write some bullet points or whatever the hell it is you're trying to say. Put it into an LLM and tell it to turn that into English text or French text or whatever it happens to be. And it will spit something out that's okay. It won't be beautiful. You're not going to come across as the next great... writer in literature, but it will be coherent. It will probably say exactly what you wanted to say. And it will actually quite importantly, it will say it in a way that most management types understand. So I would definitely start there as a techie, you know, get it, help make it help me write, write human language when I need it to, and also help get it to help me answer technical questions. Cause it will do that quicker than messing around with lots of internet search. But again, predicated on you knowing what the output should probably be approximately. I think In other jobs, it's a much harder question because, you know, if I'm an accountant, for example, can I use AI reliably and safely? Difficult because, you you've got this whole audit trail question. You know, I think, again, where you've got to write some text, it might be quite helpful. know accountants are a bit like programmers and they tend not to like writing very much. So yeah, again, that's a use case where it would probably be quite helpful. But I wouldn't, for example, want to copy and paste a balance sheet into an LLM, say, give me an analysis of this, and then rely on that answer without having gone through that whole process myself. I think that could lead to all sorts of difficulties. So it's a bit hard. I think across all jobs I can think of, using an LLM. to help you craft your written responses to things is not a bad idea. But only if you're somebody that doesn't like writing. If you like writing and you're quick at it, you'll probably do just as good a job. You'll do a better job and you'll do it probably just as quickly, ultimately, once you've been through an edit cycle. I think people are, too many people when they look at AI, because of that, it's so advanced, it seems like magic, think that... well, know, maybe they wouldn't think about it like this, but it would be, the Turing test has been passed. I'm dealing with a real thing here. Real. I'm dealing with something, an AGI and I'm dealing with something that's a person, but it's not. I mean, you can't say to an AI, hey, do my job. You know, people might think that, you know, you can do that with Claude Bort or whatever. What's it called? It's not called that anymore, it? Pardon? Yeah, that.
Shubhendu: Yeah. Okay. Full screw.
Andrew Haisley: People might think that that can do it for them. I'm sure there's some equivalents out there. Can't. You know, I mean, can make like really, really mess things up for them, but it can't take over and behave like a human would do in those situations. I don't, I think people are being hoodwinked over what this stuff can do. And it's not surprising when you have billions and billions of dollars being spent to
Shubhendu: Yeah.
Andrew Haisley: on marketing and advertising to convince them that this is the case. I think you've got to get hands-on in detail with the output of one of these things to really get a feel for what they're good at and what they currently aren't very good at. And I personally think we're hitting some, we're pretty much at the limit of what an LLM can do. I don't think...
Shubhendu: No.
Andrew Haisley: You know, it's the, when will we have AGI? another 10 years. I think we're in that stage now, right? You when we're going to have nuclear fusion, another 10 years, right? I mean, I think we'll have nuclear fusion before we have AGI, by the way, but that's a different conversation.
Shubhendu: So the honest message if I have to like synthesize what what you just said is we should learn the not learn the tool, but rather than what are the edges of the tool based on your job profile, what other things then the tool cannot do, you should know that. the limitation of that you can
Andrew Haisley: Yeah. And if you're a programmer, don't fall into the trap of using Claude for everything and getting other people to review out the problems. I this is a pattern I've heard from a few big tech companies actually, where developer will, particularly at the very junior end, having not really learned how everything fits together, will get Claude to do that piece of work for them. It will probably produce something which kind of works. And then they'll submit that as a, you you commit to whatever, you know, version management system they're using and ask somebody else to review it and the reviewer will look at it, the reviewer being a much more experienced person, they'll see just an enormous amount of code that doesn't necessarily do the job very well and then they'll go through this loop of essentially giving the person feedback, they'll put that feedback into interclaw and have Claude update its output and you go through this loop. I'm hearing that that's actually quite a common anti-pattern in larger environments and just My advice would be just do not fall into that trap. Because you'll never learn anything there. Well, ultimately, you'll probably get fired as well. It's very important that you understand what the output should look like. Then once you do that, and maybe you say to Claude, do this, you can see what his output looked like and think, it's not quite right. Give it updated instructions. It does a better job than you hand-aided the result. And possibly that took you less time than doing it yourself. I mean, particularly for...
Shubhendu: Yeah.
Andrew Haisley: things like user interface code that might work quite well because you tend to have to write tons of boilerplate and it's not concise. But anything where it's like you're dealing with a complex, highly optimized code base, using an LLN to make major changes to that is a recipe for disaster. And if you do that as particularly as a junior developer, you're going to damage your career quite badly, would be my expectation.
Shubhendu: Yeah, I I think I'll add one one thing there. a little different than what you said, but like those who are wondering where to start, what to do, the people who will win will not be the people who trust AI the most. Actually, they will be the people who know when not to. And that's very important to know the difference. There's a fine line of relying on the tool, which again using all these tools to kind of augment yourself is good, but then should know the limits of it and you should know exactly again adding to what Andrew just said that you should know exactly when not to what the what are the guidelines you should be putting in your work to kind of make sure you do not fall in that trap. But thank you so much for that Andrew. I'm sure the listeners, the viewers will appreciate that advice. Now we we do one thing every every episode, Andrew, and it this is like you asking a question for the next guest for our next episode.
Andrew Haisley: Yeah. Thank you. Yes.
Shubhendu: that next question is Michael McKernan. I think you guys have worked together in the past. so what do you want to ask him around AI?
Andrew Haisley: Yeah, we have. We have? Yes. So Michael's a very, what's the right word for it? broad-based guy, to put He knows a lot of things in lot of different domains. He's not your typical techie. My experience of working with him, which is good, by the way. I highly recommend knowing about lots of things that aren't technology as a techie. So I'm very, and something I've sort of wrestled with, particularly with the music side of things. I'm interested in his opinion of the uses of AI. in a business, professional, maybe scientific environment where possibly artistic quality isn't the number one requirement, right? Versus the use of AI in a creative environment. And, you know, my examples would be there's a lot of AI-generated music. And some of it's been quite successful. But to me, it all sounds... kind of soulless, you know, it doesn't have that sound of something produced by a person, you know, it doesn't have that, what's the word for it, torture in a spirit in it or however you want to put it. So I'm very interested in, like I said, Michael's a very broad-based guy with a lot of different interests. I'm interested in his opinion on the story, you know, where that line should be, you know, what should be. Where should we draw the line and say, yeah, AI's just don't generate good creative stuff? Where should we stop? Or maybe not where should we stop, but where does it stop? Where can people really start to tell the difference? That's my question. It's a bit philosophical, but I think you'll like that.
Shubhendu: Okay. I I think that that's a great question and I am sure it's gonna make him ponder and think a little bit. So Andrew, again, thank you so much for your time today. This is generally one of the most interesting and in like enriching conversations I've had recently.
Andrew Haisley: It's fun, I like to get to talk nonsense for a while.
Shubhendu: But I'm sure this is not the the last time we're speaking on this show, Andrew. So again, thank you again for your time. we'll be back next Monday with a new speaker for you. So stay tuned. If you like, please do do not forget to comment, subscribe. And if you have any thoughts, any feedback, anything you wanna listen to, any topics you wanna cover in in my next episode, please do not hesitate to write it down in your comments below. Thank you everyone and thank you, Andrew.
Andrew Haisley: For whom?
