speaker-0: Hi everyone, welcome to the fifth installment of Beers and SEO with Barry and Steve and I'm Steve and we've got a lot to talk to you about today, we Barry?
speaker-1: And I'm Barry. ⁓ yes, yes, we're going to be talking about AI and the impact on news and publishers with some tips and advice on how to handle optimization for AI search, if you're interested in that. And also maybe talking about AI bots, AI crawling, and what publishers might need to do to protect themselves from that.
speaker-0: Yeah, so it really is a bumper edition and every week before we record me and Barry have a list of things that we're going to talk about and I always front load it with the entrée, some things that we discussed before the main course and the main course is pretty substantial so stay tuned ⁓ and get your notepad out or ⁓ notepad LLM. Right, so Barry, first of all the most crucial item is what are you drinking today?
speaker-1: Today I am drinking Bruehaus Hoppiday. This is actually from a brewery in Belfast, in East Belfast, close to where I used to live before I moved to Cookstown. They have a lovely own brewery, they have their own tasting room and basically it's a glorified pub and a bit of a hipster place, but who doesn't like hipster places? This is a Hazy IPA and I haven't actually tasted it before, so this is my first sample of it.
speaker-0: Please feel free to complete the sample.
speaker-1: That's delicious actually. is really nice. I quite like hoppy beers and this one is hoppy without being overwhelmingly so. ⁓ this is gorgeous. I might have to drink this one more often.
speaker-0: I we need to get you onto advertising, Barry. think that's really, really good. Myself, I'm on a Northern Monk variety. So it's a classic. mean, this leads brewery. I'm probably late coming to this, but I discovered it by using one of those kind of know, beer. The convenience of beers being delivered to you. I've never really sort of like stopped appreciating that. But yeah, I've got a, I had at the time a selection of Northern Monk beers and the Eternal one. I mean, it's a session now, it's for me now. you know, I think gone are my days of the kind of like European beers where there's sort of like seven or eight percent and...
speaker-1: I can't do that anymore. Five percent is sort of my maximum nowadays and after that you just take it to piss really.
speaker-0: I mean, it was fun, fun discovering all that, but yeah, if you've got to go to work the next day, it's not so great. So, right. So the entree. So there's been like some really, ⁓ like every week, I mean, most of our jobs now are working in SEO and AI visibility, right? I mean, a lot of it is just keeping up to date with everything that that's being discussed and new reports and new analytics. It's, you know, I find my, you know, what used to be like 20 minutes before the working day, just catching up. I now have to book in like an hour ⁓ just to go through everything, particularly over the weekend when it drops stateside.
speaker-1: For me, it's about 20 % of my time is just spent reading stuff, trying new tools, staying up to date on the latest. mean, SEO has always been an industry where change is just part and parcel of what we do. You know, we always have to adapt to new strategies, new tactics, new Google features, new context research is important. But the last few years has been a bit more extreme in that regard. And I find myself having to dedicate a lot more time to just staying on top of the latest things, which I don't actually mind doing because I like reading about this stuff. I like our industry. What I don't necessarily enjoy is the stuff that was cutting at the year ago is now probably, you know, ancient history already. And you always have to redraw the lines about what advice you can give to clients. And for example, I have a rolling PowerPoint deck that I've presented several times to clients about the state of AI. and the state of user behavior and search. And if I look at the first version of the deck to the current version, it's entirely different and it's only been a year old. Whereas my standard SEO training decks, how we actually do this stuff for day-to-day optimization hasn't really changed that much, but the context in which we do that has drastically changed.
speaker-0: Yeah, and we're going to come on to a bit on that. But looking at the last couple of weeks, I think there's a of things that we might want to comment on. obviously there was the March Core update. Now,
speaker-1: Yes. We two updates. had the spam update and the core update back to back.
speaker-0: Yeah, so how impactful do you think that's been, Barry?
speaker-1: I haven't really seen significant shifts off the back of it. Neither the spam update nor the core update. know some people said they saw some shifts off the spam update and there were some minor fluctuations of the core update, but I can't really say that we can isolate the core update as being the culprit there because those fluctuations tend to fall in the regular ups and downs of the new cycle and traffic deviations anyway.
speaker-0: Yeah, I was speaking to some news publishers and without naming names, somebody said, ⁓ reporting back to senior level, some of our sites went up, some of the sites went down. yeah, we have these fluctuations, but there doesn't seem to be, I mean, I know others like Aleda and there's been many other sort of in-depth analysis on various different sort of outlooks on it. ⁓ It's always useful to look at who... winners and losers, ⁓ it's always important, but it seemed that the winners didn't win by much and the losers didn't lose by much, at least in the news publishing world. So ⁓ we'll see.
speaker-1: Yeah, fairly uneventful one, which I think we should be very grateful for. It's more or less bypassed the news industry in a meaningful fashion. So we can relax a little bit and breathe a bit easier, maybe.
speaker-0: Right. So onto our second item. this is, do you know what I normally have for like little notes to support each of the each of the talking points that we're going through. And I've just got in exclamation marks, the back button. So this is all about ⁓ Google releasing ⁓ basically that now saying that publishers who are hijacking the back button when you're visiting their sites. So by that, you know, it is a It is a terrible user experience to call that out immediately. But, you know, for those that may not have ever experienced it, you attempt to leave the site and then it will throw up an additional page of, you might also want to read this. So really it's like two clicks before you have to get out of that site. So Google are going to come down heavy on this, aren't they?
speaker-1: Yes, they've given us two months notice before they start rolling out manual penalties. And I've seen back button hijacking appear in multiple different forms, like the form you set, you click the back button, you get an overlay before you're allowed to leave the site. Other websites are even more aggressive where the back button doesn't actually go back at all, but it just sends you to an entirely different article. So basically inflating their page views by just making the back button do something else entirely as if you're clicking on a link. And I'm actually a huge fan of this particular spam policy. I have had issues with Google's new spam policies before, like site reputation abuse, I think was a massive overreach and I do not agree with it. This Google spam policy, I am in huge favor of because it is actually bad user experience. It is a dark usability pattern that some publishers, some websites have embraced as a way to artificially inflate the time on site, artificially inflate pages per visit. and has no benefit whatsoever for the user. And it's only there purely for publishers to make more money. And I think Google with its power it has over the web can make the web a better place. And it has done so before by, for example, pretty much forcing everybody to go to HTTPS, secure connections instead of HTTP. And this is one of those things where Google is using its power to actually make the web a better place. So for this one, I'm all in favor of it. Having said that, You also sort of have to question the power that Google then has over the web, that they can make these policies and basically force the entire web to adopt their rules. Because if you don't, you lose your Google traffic. And that's situation very few websites are comfortable being in. So while I like the policy, it also highlights the fact that Google probably has way too much power over the web as a whole. But you know, that's not something we want to talk too openly about, lest we bite the hand that feeds, right?
speaker-0: Yeah, which is a great song by ⁓ Trent Reznor, Nine Inch Nails. But yeah, right. Loads of SEOs and loads of publishers came out on kind of like, well, maybe not publishers, right? Came out celebrating this because, you know, just reading sort of like some of the comments around when this was announced. was, you know, some publishers argued that... ⁓ You've got to make money somehow, right? And so it's by advertising methods like this or, ⁓ know, audience, inverted commas, audience retention methods like this, you know, it pays the bread and butter for the real journalism to happen, right? But ⁓ I can understand somebody who might be in the commercial department arguing from that. But, you know, I think it's our job as SEOs or as editors, writers, is that you should always be defending the audience experience, right? Because if that's not happening, then who is?
speaker-1: I have very strong opinions on this and I have had arguments with people about this. People say, we have to do this because we have to pay for the journalism somehow. Whereas there are some websites that have gone so far with monetizing their audience that the website isn't a journalistic outlet anymore, it's an advertising vehicle. And yes, I realize we need to make money. We need to fund journalism in one way or the other. slapping these sorts of hyper-aggressive ads on it. And I include stuff that has this built in like a tabula, outbrain, web content. They have features in their software platforms that enable things like back button high checking as well. I do not agree that those platforms offer any sort of positive user experience and that if websites are... content or even happy to host these sorts of platforms and go hyper-aggressive with monetizing every single visit, it shows an inherent contempt for your audience, for your readers. It shows that you don't actually care about the usability, the experience that you provide to your audience, but that you want just to extract maximum value from your audience, no matter what the cost is to that audience, to those readers. And we have to find a bit of a balance there. Yes, we do need ads. We do need monetization. We do need to find ways to pay for the journalism. But there are things that you do that go too far. And websites that go too far are websites that, in my experience, tend to lose in the long run because people start building these subconscious associations that that website is not fun to use, not pleasant to use, and people will try to avoid going to those websites. Whereas I think if you're still focused on providing an optimal user experience within the context of what you can do with monetization, you tend to fare better in the long run. You find it easier to build audience loyalty. You find it easier to maybe to get people to subscribe, to ⁓ pay for your content, to register for registration rules, all those other things that you can do to make loyal audiences. So I think publishers need to be very aware and push back on their own commercial departments. that they don't go too far with injecting these nasty dark usability patterns that serve no purpose at all except to maximize money and decrease the actual value of your website to your actual audience, the actual readers that consume your content.
speaker-0: Yeah, I guess it's, it's really down to if audience teams or, or data teams or UX, you know, product teams, if they're empowered and have got an equal, ⁓ know, equal level with the commercial team. Cause you know, sometimes it may just be that the commercial teams, they've got like the, primary communication channel with the powers that be. And, you know, you really need to be listening to the other specialists in the room that really. No, the audience,
speaker-1: It's a leadership challenge. Leadership needs to set these boundaries and needs to build that internal company culture that makes sure that everything that the company does to improve and to grow doesn't go too far in building bad experiences for their audience.
speaker-0: Okay, agreed. So yeah, so really there's been a couple of reports particularly in the New York Times and saying which Lily Ray is also commenting on around AIO ⁓ accuracy or lack of accuracy. Do you want to talk a bit about that?
speaker-1: Yeah, there was a New York Times article where they did a fairly small study that showed that about one in nine, one in 10 AI overviews was inaccurate to various different degrees. And they wrote an interesting piece about that. Google fired back and said that these were not representative searches part of the study and they were deliberately crafted to elicit bad answers, inaccurate answers. And Lily Way responded to that. Google response saying, actually no, these are fairly representative. This is a normal user behavior or fairly reflective of normal user behavior. And the accuracy rate in AI overviews is about 90%, which means that yeah, one in 10 AI overviews will have a bad response in it somewhere, which shouldn't really surprise anybody by the very nature of LLMs. The hallucination problem is not solved, can never be solved, it's inherent in the technology. So of course, AI overview is going to be inaccurate. But Google wants to safeguard its image. And personally, I don't think the media make enough of the fact that AI overviews are inaccurate. They don't show enough examples. And people just take these AI generated answers as gospel and make very important decisions sometimes. based on the information provided to them by an AI overview or by a chat GPT answer or by a perplexity answer without necessarily double checking it. So I think we probably need to give this more attention and tell people, okay, it's very useful. It's very handy getting an answer from an AI, but at the same time, double check it. Yeah. Especially if you want to make any important decisions based on it regarding your finance, your health, your relationships, and don't accept it at face value ever.
speaker-0: Right. I mean, it's all about regulation, right? mean, it's like using a food example, a fast food example, right? It's like to put clearly label. it well, maybe not food, maybe cigarettes, right? It's kind of like, that took many years to put that kind of warning on it. But it feels, I mean, we're sitting now looking at how the search results look like, which we're familiar with for many years, but... Across all these services there should be some kind of warning, Or treat with caution.
speaker-1: Yeah, a health warning. I also see the other side of the coin here where if you go to a social media website, ⁓ TikTok, Instagram, Reddit, YouTube, and you see something said there by a human, you tend also not to necessarily take that at face value, at least you shouldn't. The difference is of course that we have been trained by the last 25 years of Google to sort of trust their... responses, that there's a certain level of authority built into what Google shows you. with AI especially, with the confidence that AI presents its answers, it tends to really throw people's bullshit detectors off and they tend to just say, ⁓ it's set with such confidence and it comes from Google, therefore it must be true. And I think that's something that we need to maybe teach people how to handle better. that okay, even if it's presented with confidence by a trusted brand, does it necessarily mean it's actually true? And you might want to dig a little bit deeper before you do anything with that information.
speaker-0: Critical learning, critical skills, we all need to sharpen up on them, right? So, I mean, that brings us directly into AI, right? Into the main course, if you like. So, you know, it's been a huge learning curve for myself and I assume you, as you've said, Barry, you there's something new to learn every day, but just from the top of my mind, I saw a post recently which didn't really comment on the latest report or the latest findings. was just asking a selection of 10 to 15 experts ⁓ how they are coping with all the reports which are coming out. Because even if you thought like, I remember when I first started, I was studying SEO, studying ⁓ Google. You know, there is no, I mean, obviously you can read Google documentation, right? But there is no central resource where you can learn everything. It's a much more complex thing. You've got to, you've got to read the latest studies, you've got to network a lot. You've got to kind of like, you've got to test, see, you know, Google might say something, you've got to put it into play and see what actually happens. There's a, it's a holistic approach to it, right? And now you've got ⁓ AI visibility and someone is telling you every hour, every minute on LinkedIn. you need to do, right? So that's what we're here to kind of like, kind of give you some guidance on that. And with that, I'll hand over to you, ⁓
speaker-1: I think there's two things you want to cover today, which is ⁓ AI crawlers, AI bots and scrapers, and how publishers should maybe interact with those, and then talk about whether optimizing for AI is a fool's errand or maybe worth doing in the first place. I want to start with crawling and how AI crawlers are currently behaving, because that's also a bit of hot topic at the moment. is also an accelerating topic because the AI bots are getting much more aggressive. I've seen some interesting data, interesting reports, as Perlman just said, because this stuff just keeps flowing out at like one report a day and it's hard to stay on top of. But this was an interesting one from Tolbit, which has a state of the bots report that they do every quarter. And for the last quarter of 2025, they had an interesting report that showed that There's very aggressive crawling happening by AI bots, primarily by GPT. And most of that is not actually training data. There's two types of crawling. Generally that happens, increasingly three types, but let's talk about the big two types, which is a calling for training data, which is how AI tools gather content to either train an entire model or fine tune and improve an existing model. And then there's ⁓ retrieval augmented generation, RAG. which is when someone actually does a prompt on an AI system like ChatGPT and ChatGPT goes in and basically searches the web for relevant pages and then retrieves them and summarizes them and provides them with an answer and shows them as citations. The training data crawling is less than one-tenth of the retrieval augmented generation crawling, whereas it used to be the other way around. But that also has to do, of course, with how many more users have now adopted ChatGPT and other AI bots. So there's much more usage. So there's less need for AI tools, LLMs to train their models. They already have the data to a large extent and just keeping those training data up to date isn't that intensive a process anymore. Whereas the sheer use case of AI tools means that there's tens of thousands, hundreds of thousands of prompts happening every minute. And therefore there's a lot of retrieval, augmented generation, calls being done. And then there's a third layer of calls, which is the AI agents, but it's still a very early stage. It's not a huge activity there for now. So we don't really have to talk about that too much. What I find really interesting when I look at those sorts of reports and the amount of calling that's happening in that it tends to outpace the calling of classic bots like Google bot and Bing bot. And more importantly, That call behavior from especially chat GPT does not seem to align to the value that the caller generates for the website. As in yes, okay, you get called 150,000 times by chat GPT and you get 12 visitors. Whereas if you get called 150,000 times by Google bot, you get a couple of thousand visitors. Usually the value exchange is very different, which makes it very worthwhile for websites to allow Google to call them and index them and use their content. Whereas the value exchange for ChachiPT is less clear. It merely means what the goal is of you being shown in a ChachiPT generated answer. We can talk about that a little bit later as well. And you have methods of blocking certain crawlers on your website, but those methods are not necessarily foolproof because there's certain workarounds that the AI tools use. Again, to stay with ChachiPT for a moment, they have The main retrieval augmented generation caller is ChatGPT user, which is what they call user-instantiated, which means they only trigger a call when a user asks for a specific prompt. And that also means that ChatGPT feels it shouldn't necessarily respect your robots.txt disallow rules. Because when you browse web pages in a browser, you also don't respect robots.txt. If there's a link to a page, you can just visit it in the browser. whether or not a bot like Googlebot has access to it. And ChatGPT extends that to its ChatGPT user crawls as well, because the user instantiated, they basically see it as sort of a browsing session, and therefore should be able to load the webpage. Of course, there isn't actually a browsing session. The page doesn't actually load. It's just ChatGPT downloading the HTML, extracting the content, regenerating it in an answer and serving it up to their users. So again, there's no value exchange. You don't get a visitor to your webpage, whereas in a browser, of course, you do. So I think that's a little bit of a sneaky method. ⁓ But you can still probably block chat you with a user if you want on a hosting level. But then there's an extra layer, really, really problematic layer, I think, is that there's a lot of third party crawlers, like a lot of them, that will basically scrape the web, everything they can find. And using user agents that you probably have never heard of. And they probably change user agents all the time as well to mask themselves and become different crawlers in your log files, but basically all the same crawlers. And they gather all the data and they then sell it to AI companies. And there's a lot of example of that. There's a few posts on LinkedIn I saw about this as well.
speaker-0: That's a key thing, right? You may have some established AR companies who say we don't do this, but then there's the acquisition of that data. did they get the data from?
speaker-1: The AI company says, yeah, you can block us in robots.txt and we won't call you. Great. But then they still get your data by just buying it from other crawlers, from third parties. So you're not really blocking anything. again, the crawling behavior is very aggressive, tends to ignore robots.txt also because most robots.txt files will not have specific block rules for those user agents because they tend to change user agents all the time or they might not even know that they exist. And some of them just plainly ignore your OSTT file anyway, because they don't care. There's no legal repercussion. They just ignore your block, your digital rules and call you anyway.
speaker-0: Yeah. And I found, you know, some clients have come to me, they're quite confused about what they should be doing with the robots TXT file. Right. So, you know, it's, you know, it's a, it's not a directive. And as you say, there's many ways that the bots can bypass you. But I still think it's a useful thing for publishers to do, you know, for those that do not know, please go and check out the robots TXT file. So it's literally domain, forward slash robots dot txt. ⁓ many publishers, particularly who might be on WordPress or other kind of out of the box, ⁓ CMS solutions may never in their lives have to look at a file like that. But, ⁓ you know, that's where you, that's why I think it's a really good, ⁓ discussion to, ⁓ for publishers who may not have, who may not have explored what they should be doing. with AI bots is to start somewhere and starting is looking at your own robots.txt file or go to a publisher like the New York Times, a well-established publisher, look what they've done and try to explore why they've taken those decisions. That doesn't mean that you copy and paste because there's the key decision-making beside each disallow entry in there. But I found it a useful discussion when there's so many options. Well, let's just start with this. then it's ⁓ then by discussing what's the difference in the bots, which are training bots and which are the ones that we want to keep to our site like Googlebot, right? We don't want to disappear from the SERPs. treat with caution again.
speaker-1: And I've always been a fan of just allowing the bots to crawl and then disallowing the ones that don't behave or that you don't feel have a positive value exchange. But now I actually think we probably need to flip it around because bot behavior is so aggressive. think more than half of the web traffic is now just crawlers and bots. And the bots are so aggressive and basically devious to a large extent that I think we now just need to block them all by default. And yes, do that in robots.txt and then... have allow rules for the bots you do want to crawl. Google bot, Bing bot, some specific other bots that you think there's an added value there. And then just block everything else. And that will actually stop most of the bots. Because most bots too tend to be polite crawlers in the sense that they will adhere to robots.txt dissallow rules. But there's also plenty of bots who just ignore it and crawl anyway. Which is where hosting level blocking comes in, which is not necessarily Well, we're still TXT, but your server, your hosting environment, your CDN detecting bot crawls and saying, you know what, your bottom can block you unless you're specifically allowed. And CDNs like CloudFlare already have these things enabled. And I think more CDN providers will follow. And again, it's one of those symptoms of how the web has changed from an open free access ecosystem to increasingly walled gardens. This is all another extension of that where, you know, and I sound maybe a bit old fashioned or slightly depressed of this, because I remember these old days of the web, Steve, like you do, where it was just a free fall. was like the mild west. could do anything. Launch it and it was good. It was happy. ⁓ If a little bit of a cowboy environment as well. Whereas now everything is going behind walled gardens, paywalls, logins, you name it. And even... you know, access to content by machine systems is now being blocked very aggressively on the hosting level because we have to, because it's self-defense, pure protection, because the economics of the web have now been skewed in such a way that giving away your content for free without any safeguards is Not commercially viable in any way, or form. Unless you're happy to give it away. And there's plenty of people who perfectly happy with giving it away. But if your business revolves around the production and distribution of content, you need to have safeguards in place. And hosting level blocking is the next step. After payrolls, after registration rules, all those things, we now also have to block bots in a very direct, aggressive way on a hosting level. And only I feel allow... bots to call your website where you know you have at least a good chance of a positive value exchange. And that means search engines like Googlebot, and we can complain about Google's diminishing traffic to the web all we want. It is still by far the largest traffic source to the web with no real competition. So yes, you should let Google call your content and index it to show it in search results, to show it in the discover feed, things like that. Some other smaller search engines being bought, let's go and do that Yahoo, whatever other qualifiers are, DuckDuckGo, Baidu if you're in China, great. And then when it comes to the AI systems, you need to have that internal debate whether you want to be part of the training data and be shown as a cited source in their answers, which retrieval of many generation, whether you just want one or the two, no training data, but we do want to be cited as a source, or if you want none of it, and then block all the AI bots by default.
speaker-0: Right. mean, just on that point about being in training data, right? I mean, that's not a quick fix, right? So, ⁓ you know, so if you're in the training, if you're in training data, you've almost got like a pole position, right? Or visibility in there.
speaker-1: Not necessarily. The training data is useful to a certain extent, but because nearly all the prompts now also rely so heavily on retrieval augmented generation, the training data is basically just there to determine the quality of the model in the first place. How good the LLM model is in terms of the quality of its output, the accuracy of its output. I'm actually more okay with being part of the training data than part of the retrieval augmented generation nowadays, because the training data, just because the LLM model is out there, the technology is there. And we want these tools to be as good as they possibly can be, but not putting Pandora back in its box. okay, let them be trained on my data. I think being part of the training data also means that you can build associations, that if you have a strong brand in a specific area, for example, SEO, and you're mentioned in the context of SEO very often, I think in that initial ⁓ generated answer that is not augmented, that is purely constantly LLM, there's a higher chance of you being cited in it because that association already exists. Then the retrieval of the generation where the call or ⁓ the LLM actually goes and has a generated answer, let's see how accurate it is, or some LLMs do it the other way around. They start by retrieving and then summarize it. That's a different sort of value exchange. Then you have to decide, right, it's a lot of call activity on my website. There's a lot of load being placed on my server. by these retrieval systems to get me a mention in an answer in some context or another. And then you have to decide, is that worth it? Can I attach a value to being part of that answer that has a positive economic value for me somewhere down the line? And that's a much more difficult question to answer at the moment.
speaker-0: Yeah. And you know, it's really interesting what you say because I was reading an article in Digiday by Sara Gialoni, who was writing a response piece to this information out there, that third party bots are scraping a website site mad. She had this great quote, which I've got here, which is, said, most brands have no idea whether they're being mentioned or ignored by Chatjibi Tea, Gemini and Perplexity when users ask questions in their category. There's no rank tracker. for AI generated answers, no impression data, no notification when a competitor gets recommended instead of you. The result is that most content teams are investing in AEO based on theory, not measurement. They're publishing content, optimizing for AI search and hoping it's working with no reliable way to know whether it is.
speaker-1: Yes. I mean, there's a lot of talk about prompt tracking and its accuracy and the truth is prompt tracking is just not accurate in any way, or form. There are some studies on this actually that show that you can do the exact same prompt 10 different times in the same AI bot and you get 10 different answers, 10 different set of citations, 10 different brands mentioned. Now there are some consistency signals there for some brands, especially brands that have very strong positions in their market. And therefore they will always be mentioned in one way, shape or form. as part of a particular answer. But prompt tracking in itself, I feel is such a colossal waste of time and energy because it doesn't really give you meaningful data. In fact, it gives you bad data and bad data, in my opinion, is worse than no data at all. So I think, you know, if you're currently paying a lot of money for prompt tracking and whatever shape that that takes for you, I recommend stop that. Don't do that. Maybe do some experimental prompts now and again if you want to show some data to some senior level executives in your company. But as a whole, the exercise is fairly pointless because of the inherent instability of an AI-generated answer.
speaker-0: Right. right. So, mean, other options, we will go into a list of things that you can do from the skills that we have between us. But I would always call out, you know, ⁓ obviously having a, ⁓ having a view, a strategy, an AI bot strategy is obviously a key thing for publishers to do. ⁓ but it's not the only thing they can do. And so I would always argue that they should, wherever possible. publishers in general should join part of their collective bodies to have more of a clout. Now, ⁓ there are many options out there. We've got the PPA, the AOC, you've got the Newspur Collective with the likes of the Guardian, Financial Times, etc. All these are great steps forward. ⁓ But I wanted to recommend here a Ricky Sutton's podcast, Future Media. where he was ⁓ speaking to an expert that was previously a Danish MP who is part of a new collective there. It's a really fascinating podcast because it just demonstrates the power of working together as a collective. There are obviously less publishers in Denmark, but they've all got together. and ⁓ they're making a difference, partly because of how the political system is set up, but there just seems to be a moral agreement among them that we can't have this misinformation ⁓ and our business is going to go if we don't do something. So we need to stand up together as a collective to do that. So I think we can all take a lesson from that. And that was a great listen to.
speaker-1: Absolutely. The classic tactic of divide and conquer is too powerful. And the only way the industry as a whole can get meaningful change done is by combining forces, which is one of the reasons why I like this pro-coalition here in the UK so much. Where they have such massive media brands associated with it. And I think that'll be absolutely key to get government regulation in place to protect intellectual property, to protect copyright. and protect the sanctity of journalism in this country.
speaker-0: Right. So, do you know, one of the things I wanted to pick up on is everything that you've just said about the low value exchange for publishers, which, you know, we obviously get, particularly in news, because what you click through is the product, right? So if that's not leading to a subscription and per click that it's kind of like low value anyway. ⁓ You know, it just taking, that away from publishers is a real kick in the teeth. But, um, you know, when you kind of look at LinkedIn and other sources of information and you see the big evangelists of, uh, uh, prompt tracking or just AI visibility in general, I often think, well, actually that's because they are connected more to a brand, uh, capacity, right. Or to like, like an automobile, right? So if you're, if you're selling 30 grand. 50 grand, 100 grand cars, then obviously you don't want to lose any position. you know, that's why you're seeing in brands, you're seeing a lot of teams being ⁓ built in house, right? Where there might be more of an argument to battling against competitors in that space. But obviously, you know, if you talk to most news publishers, there doesn't seem to be this kind of... assortment of teams assembling ⁓ to build AI visibility teams, right? It seems to be more something that SEOs or audience teams are kind of looking at. I mean, would you agree just because the value exchange is so low, that's why it's not happening?
speaker-1: Yeah, yeah. If you have a direct sale opportunity or even an indirect sale opportunity and part of a customer journey through a funnel, then being visible in AI-generated answers has value because people do use it to do research. ⁓ Sticking with your car example, you can imagine people asking what is the most reliable car, what is the best value for a money car, what is a ⁓ fairly good affordable luxury car, things like that. And if your brand matches those sorts of queries, you want to be shown in those answers. So optimizing for AI visibility in those contexts, very important. Don't underestimate it. You probably want dedicated teams focusing on that. Even if you can't directly measure it. But that's the same way we have it classic advertising. You you don't know what part of it works, but you know that it works, so you just keep doing it. And the same with AI visibility, you sort of need to see it as indirect advertising in AI. If, however, your output your product is content, you have a very different context when it comes to AI visibility, because the output of AI visibility is content. That's what they do. So if you are part of an AI generated answer as a content website, there is no need for the user to go beyond that. There is no secondary conversion down the line. They just interact with your content through the medium of an AI generated answer. and there is no need for them to visit your website, even if you're cited as a source. If you are a traffic-dependent website, which most publishers are, there is no value, none whatsoever, in being visible in an AI-generated answer, because the traffic just isn't there. Even the highest click-through rate we see on AI-generated answers, which is in Google's AI overviews, It's 30 to 40 % lower than a classic search result. Now that still means that's traffic potential and there probably is still some value exchange being shown in an AI generated answer on Google. But then if you go to the next big boy, which is of course, ChatGPT, it sends almost no traffic to the web. In fact, I think ChatGPT a few months ago even changed its citations and and external links in its answers to keep people on chat GPT more often because they felt too many people were clicking through. The highest percentage of traffic I've seen to a news brand from chat GPT was I think the New York Times, and this is public data so I'm not giving away any secrets because I don't have access to the private data anyway. But I think it's like less than 1 % of the total referral traffic comes from chat GPT. And I think on average, New York Times like 30, 40 % comes from Google sources. So we're talking like an order of magnitude of for variance there. So that makes it a very interesting conundrum for publishers. And then they have to maybe look at what secondary benefits there might be to still be incited as a response in various contexts. You can imagine, for example, a website like the Financial Times, which is a very hard paywall website, that they still probably want to be shown in certain contexts for financially relevant queries in these AI tools. because it also shows them as a trust brand, as in, hey, we have this information, you want to subscribe to us, you want to come to us if you want to directly from the source. And that works if you do have that higher level of brand perception or brand power and a product that you can sell in association with that, which in the Financial Times case is the hard paywall content. But if you're more of a commodity news website and you have fewer distinguishing content types, It's very difficult to justify optimizing for AI because there doesn't seem to be any value in it for you as a publisher to do that.
speaker-0: Yeah. I mean, ⁓ it's only going to be the bigger publishers who are going to say, we're going to invest in this because we want to fight misinformation, right? Because they've got the capacity to do it. There might be other reasons like, mean, ultimately it's, you know, a lot of AI visibilities around brands, reputation, right? Who's citing you, who's quoting you, what the other's saying about you. Okay. That's popping up in AI. We need to, we need to look at that, which you could argue we've done many times before in SEO. So I'm not just not going to that, but, ⁓ you know, I think we want to kind of like move on to kind of like things that we can do from a technical SEO to, help publishers. Right. So we've already had a comment podcast before, even in our first podcast, which was around, ⁓ you know, at the, at the source, there's, ⁓ there's a lot that you can do in technical SEO in order to advance your, your AI visibility. ⁓ but We'll just kind of like go through a couple of them, right? So I think we've got semantic HTML.
speaker-1: Yeah, semantic HTML specifically for retrieval augmented generation, it allows an AI system to very easily identify boilerplate content versus core article content and use that core article content then to be ⁓ used as a citation in the AI generated answers.
speaker-0: Yeah, we've also got, I mean, this is a classic one, subheadings as questions, right? I mean, we've done a lot of that, particularly not in explainers for SEO, but these pieces still, ⁓ they appear a lot. I mean, on that, it's interesting about evergreens, right? Because some publishers have pulled away from doing that, but ⁓ it's a way to demonstrate added value, right? To the users, because you're... ⁓ First of all, you're listening to the audience about what it is that they want to know, and you're putting it in one place, which is added value, sitting alongside the actual news story.
speaker-1: Exactly. I think when you understand the questions that your audience asks and you put those questions in your actual content and then answer them in the subsequent paragraphs, you're helping your users because they just get the direct answer to what they're looking for. And you're helping the LLMs because they know, ⁓ you've got the answer to that particular question that the user has asked. So I'm going to use your content as a citation.
speaker-0: Yeah. And structured data, right? That's been sort of like debated because I mean, this is why everything flip-flops, you know, from day to day on what's the general consensus on this. But I think we got into some position at some point where some practitioners were saying, ⁓ you know, if you just concentrated on unstructured data, that's definitely going to advance your mentions in your citations. And then others were saying, well, wait a minute, what is actually the content on the page, right?
speaker-1: Yeah, the jury is still out on that one. There's one particular context where we know structured data helps and that's agentic optimization. If you want to optimize for AI agents interacting with your content, they tend to look at structured data because it's basically a machine readable summary of your web page and allows them to extract the right information, make the right actions. For pure content-driven AI citations, the added value of structured data is not that clear. But structured data generally has positive effects, especially if it's a schema type that Google supports. So always a fan of implementing it when you can.
speaker-0: Yeah. Okay. So chunking, right? So chunking ⁓ has been, I mean, we're in this interesting environment where we're all learning about LLMs and then anyone that says or identifies a pattern will come out and say, you need to do this. then, you know, the industry is just too young to latch on to anything with any concrete evidence, but we're still there. looking for the main trends, the main patterns, right? But it's interesting about this element of chunking. So looking at paragraphs, ⁓ even that has come under fire. you know, even Google spokespeople have said, you know, it's not a long-term strategy to chunk everything. However, from my understanding, you know, what chunking actually indicates is really that ⁓ you should look at line by line. So really for AI visibility, it's all about just getting to the point. So now this is interesting a point, right? Because you've got to write for users and write for the bots and what do you do? We're always looking for a happy medium between the two, but sometimes I want to read and I want to subscribe to copy which sends me on a journey, right? So often opinion writing or commentary or just humor writing, that is, comedy writing. I'm sounding like a bot now. It's meandering and like I'm talking now, just, whereas for AI bot optimized copy, it's very much line by line ⁓ optimization. Get to the point, put the main facts in, the main entities, move on to the next point. No dilly dallying. that's the kind of approach that you take and you know there is an art in that ⁓ but yeah it may be less so for news publishers.
speaker-1: Yeah, the chunking is always a bit misunderstood, I think, by many SEOs. I don't pretend to understand it fully myself, but I do know that in the context of LLMs, it's about the context window that an LLM has to understand a sentence or a paragraph, which is not as big as people think. It's not a huge context window, which is why what you just said, get to the point, know, short, concise sentences, that is what the LLMs want, because it allows them to understand the point of your sentence in that small context window that they have. but it doesn't make for pleasant reading. It doesn't take you on that journey. Which is why I am a strong fan of not applying this as a direct tactic for human focused journalism, but to write for humans, to craft that narrative, to build that story and just make sure it's pleasant to read and engaging with the article is pleasant for your users. I think that is a much better long-term tactic than trying to artificially... break your content up into short snappy sentences that are just not fun to read.
speaker-0: Yeah. And also, there's also a side note here on using, actually, when you're using AI to help create copy, just be very careful. I think the general consensus there is, you know, it's fine to have AI assisted copy to help you. But if you are using AI to just generate the actual idea of the article, be very careful of that because it's only sourcing what it's already read, right? So will it mention you if you're just redoing something it's already got? So it's definitely worth being careful there. This is an old favourite of mine, crawlability and crawl depth and internal linking, right? Yes. I think that's our speciality, isn't it? Yes.
speaker-1: I think there's a lot of this is actually a lesson that publishers can embrace because not a lot of publishers allow deeper easy access to the article archives. I think that's a missed opportunity in the context of search to a certain extent, but especially in the context of AI, where every article that an AI has access to both for training data and for retrieval of mental generation improves your presence in their systems. And there's too many publishers that just have like a category page with 20, 30 articles and that's it. And maybe there's some pagination if they're well set up to maybe 10 pages of articles, which again is fine for users. But there's very few publishers who allow deeper access to older content with archive links, HTML sitemaps, things like that. And I think that's probably a missed opportunity for many publishers. If you are a high quality publisher and you have a whole archive of tens of thousands, hundreds of thousands of good journalistic articles, you should probably build a pathway. for crawlers to find that in a very easy way through a date-based archive or some sort of HTML sitemap. So that it's only a few clicks away from your homepage. One example I will quote for this for a website that I think does this really, really well is TechRadar. TechRadar has like nine pages of pagination for any one of the tags and categories. And then there's a link to their archive, which is a unified archive, date-based, which basically allows you to find every article published on their website, on the techradar.com website, in its entire history with maybe three clicks. And I find that a very elegant solution. Some people worry that that sort of solution creates a lot of crawl overhead for Googlebot. It really doesn't. Googlebot knows to focus its crawling generally on new pages that it hasn't seen before. And we'll call the old pages once in a while to check that they haven't changed, but usually only when it has crawl effort to spare. So we shouldn't worry too much about that. And I find that sort of a solution a very good way to enable that all the content to be called and indexed and also to be visible to the AI box.
speaker-0: Yeah, it's interesting because when I've looked at particularly news publishers who have visibility in AI and chat, GBT or perplexity and the like, I've often noticed that it's links to the archive. Right. So that, mean, it's an AI bot dream, right? You've got everything there for it and it's easily accessible. So yeah, I've noticed brands like the Independence get great visibility through that. So yes, go and have a look at the archive system ⁓ on publisher sites. Okay, MetaTags and OpenGraph.
speaker-1: Yeah, still used by both search engines and AI bots, know, the title tag, canonical tags, meta descriptions, open graph title, open graph description, open graph images even. Because when a chat GPT shows a citation for a webpage, it generally has a thumbnail image associated with it as well. And that stuff is just classic SEO as well. You want to make sure that they will find in an order and, you know, properly optimized to give meaningful descriptions and titles for your content.
speaker-0: Okay, I agree with that. obviously with titles, particularly around optimization around Google Discover, right, which is still, we know, big.
speaker-1: Open Graph especially. think the last date I saw was that 60 % of Discover headlines come from the Open Graph tag, not the title tag. So that's an interesting one to keep.
speaker-0: Yeah. Okay. LLMs, LLM text.
speaker-1: Yeah, again, a jury is still out. I do see data that some crawlers, especially chat GPT, do tend to call LLMs.txt files with some regularity, but it's not on officially accepted standard yet. and most of the implementations come off the back of just basic default functionality of like an SEO plugin on WordPress. So, publishers might not even be aware that they have an LLM.txt file. If you don't have it enabled at the moment,
speaker-0: Yeah.
speaker-1: I do not necessarily see value in creating one just yet. I think we need to wait and see if it actually becomes a standard for LLMs to use it. At the moment, I don't see that there is a good business case for it yet.
speaker-0: Yeah, mean, like there's even this week there was ⁓ a discussion around the benefits of doing that. But for the last year, there's been a consensus of not really doing it. again, literally things can change within days of in this fast moving world of AI visibility. I think we've covered a lot of topics there.
speaker-1: I want to talk about one more thing because it's a bit of hot topic as well at the moment, which is Markdown. Markdown is basically a stripped down version of the content of your website served in machine readable format, which you can serve hypothetically to AI crawlers so that they don't get your full website, they just get the Markdown version of your website. And that is seen as a way to basically present an AI crawling optimized website to crawlers from AI machines and present new regular users with
speaker-0: Go for it.
speaker-1: normal web pages. But again, it requires development effort, requires investment and resources in that technology. And the benefit to cost ratio isn't really there because LLMs have, since their inception, been very, very good at just calling HTML pages, standard web pages. That's what they're designed to do, to basically separate out the boilerplate content, separate out the HTML tags and look at the actual content of the page. They don't need markdown. They really don't. And what you're doing if you're giving a markdown version of your content to specific crawlers and a standard version to users, you're falling into the same trap we fell into with accelerated mobile pages. If you remember that AMP, you basically have an entirely duplicate version of your website for one specific user context. And that just creates an entirely separate development, parallel development track for your website. That is a ⁓ potential source of all kinds of conflicts and problems. And especially in the context of markdown, there isn't really a value exchange there because the LLMs already are very good at parsing HTML. So I think when it comes to markdown, that's a big no at the moment and really not worth the investment.
speaker-0: Right, steer clear. Okay, right. So that's a lot to consider. I think that was a hearty meal this time. Okay, right. So I think we're going to sign off there. Obviously, ⁓ we would always cherish feedback and comments and recommendations for what to cover on this podcast. We will be attending Brighton SEO in a couple of weeks, won't Barry?
speaker-1: Yes, we are. I think two weeks from now, two weeks and two days. I look forward to it. I tend to only go to the spring edition rather than the autumn edition. So the spring edition is coming up. And for me, Brightnessio is all about the people who are there and the connections that you make and catching up with old industry friends. So I look forward to seeing you there as well, Steve.
speaker-0: Yes, I'll be there with ice cream in hand, although I think last time it rained. But anyway, we'll be down on the beach, our toes in the water, maybe with a beer in hand. But like, please remember to, I'm going to do that thing of please comment, like, subscribe on our podcast, wherever you can find it. We're across all platforms. And like I said, really would look for recommendations to discuss on future episodes. But for now, thank you for listening.
speaker-1: Thank you very much, everybody.
speaker-1: And I'm Barry. ⁓ yes, yes, we're going to be talking about AI and the impact on news and publishers with some tips and advice on how to handle optimization for AI search, if you're interested in that. And also maybe talking about AI bots, AI crawling, and what publishers might need to do to protect themselves from that.
speaker-0: Yeah, so it really is a bumper edition and every week before we record me and Barry have a list of things that we're going to talk about and I always front load it with the entrée, some things that we discussed before the main course and the main course is pretty substantial so stay tuned ⁓ and get your notepad out or ⁓ notepad LLM. Right, so Barry, first of all the most crucial item is what are you drinking today?
speaker-1: Today I am drinking Bruehaus Hoppiday. This is actually from a brewery in Belfast, in East Belfast, close to where I used to live before I moved to Cookstown. They have a lovely own brewery, they have their own tasting room and basically it's a glorified pub and a bit of a hipster place, but who doesn't like hipster places? This is a Hazy IPA and I haven't actually tasted it before, so this is my first sample of it.
speaker-0: Please feel free to complete the sample.
speaker-1: That's delicious actually. is really nice. I quite like hoppy beers and this one is hoppy without being overwhelmingly so. ⁓ this is gorgeous. I might have to drink this one more often.
speaker-0: I we need to get you onto advertising, Barry. think that's really, really good. Myself, I'm on a Northern Monk variety. So it's a classic. mean, this leads brewery. I'm probably late coming to this, but I discovered it by using one of those kind of know, beer. The convenience of beers being delivered to you. I've never really sort of like stopped appreciating that. But yeah, I've got a, I had at the time a selection of Northern Monk beers and the Eternal one. I mean, it's a session now, it's for me now. you know, I think gone are my days of the kind of like European beers where there's sort of like seven or eight percent and...
speaker-1: I can't do that anymore. Five percent is sort of my maximum nowadays and after that you just take it to piss really.
speaker-0: I mean, it was fun, fun discovering all that, but yeah, if you've got to go to work the next day, it's not so great. So, right. So the entree. So there's been like some really, ⁓ like every week, I mean, most of our jobs now are working in SEO and AI visibility, right? I mean, a lot of it is just keeping up to date with everything that that's being discussed and new reports and new analytics. It's, you know, I find my, you know, what used to be like 20 minutes before the working day, just catching up. I now have to book in like an hour ⁓ just to go through everything, particularly over the weekend when it drops stateside.
speaker-1: For me, it's about 20 % of my time is just spent reading stuff, trying new tools, staying up to date on the latest. mean, SEO has always been an industry where change is just part and parcel of what we do. You know, we always have to adapt to new strategies, new tactics, new Google features, new context research is important. But the last few years has been a bit more extreme in that regard. And I find myself having to dedicate a lot more time to just staying on top of the latest things, which I don't actually mind doing because I like reading about this stuff. I like our industry. What I don't necessarily enjoy is the stuff that was cutting at the year ago is now probably, you know, ancient history already. And you always have to redraw the lines about what advice you can give to clients. And for example, I have a rolling PowerPoint deck that I've presented several times to clients about the state of AI. and the state of user behavior and search. And if I look at the first version of the deck to the current version, it's entirely different and it's only been a year old. Whereas my standard SEO training decks, how we actually do this stuff for day-to-day optimization hasn't really changed that much, but the context in which we do that has drastically changed.
speaker-0: Yeah, and we're going to come on to a bit on that. But looking at the last couple of weeks, I think there's a of things that we might want to comment on. obviously there was the March Core update. Now,
speaker-1: Yes. We two updates. had the spam update and the core update back to back.
speaker-0: Yeah, so how impactful do you think that's been, Barry?
speaker-1: I haven't really seen significant shifts off the back of it. Neither the spam update nor the core update. know some people said they saw some shifts off the spam update and there were some minor fluctuations of the core update, but I can't really say that we can isolate the core update as being the culprit there because those fluctuations tend to fall in the regular ups and downs of the new cycle and traffic deviations anyway.
speaker-0: Yeah, I was speaking to some news publishers and without naming names, somebody said, ⁓ reporting back to senior level, some of our sites went up, some of the sites went down. yeah, we have these fluctuations, but there doesn't seem to be, I mean, I know others like Aleda and there's been many other sort of in-depth analysis on various different sort of outlooks on it. ⁓ It's always useful to look at who... winners and losers, ⁓ it's always important, but it seemed that the winners didn't win by much and the losers didn't lose by much, at least in the news publishing world. So ⁓ we'll see.
speaker-1: Yeah, fairly uneventful one, which I think we should be very grateful for. It's more or less bypassed the news industry in a meaningful fashion. So we can relax a little bit and breathe a bit easier, maybe.
speaker-0: Right. So onto our second item. this is, do you know what I normally have for like little notes to support each of the each of the talking points that we're going through. And I've just got in exclamation marks, the back button. So this is all about ⁓ Google releasing ⁓ basically that now saying that publishers who are hijacking the back button when you're visiting their sites. So by that, you know, it is a It is a terrible user experience to call that out immediately. But, you know, for those that may not have ever experienced it, you attempt to leave the site and then it will throw up an additional page of, you might also want to read this. So really it's like two clicks before you have to get out of that site. So Google are going to come down heavy on this, aren't they?
speaker-1: Yes, they've given us two months notice before they start rolling out manual penalties. And I've seen back button hijacking appear in multiple different forms, like the form you set, you click the back button, you get an overlay before you're allowed to leave the site. Other websites are even more aggressive where the back button doesn't actually go back at all, but it just sends you to an entirely different article. So basically inflating their page views by just making the back button do something else entirely as if you're clicking on a link. And I'm actually a huge fan of this particular spam policy. I have had issues with Google's new spam policies before, like site reputation abuse, I think was a massive overreach and I do not agree with it. This Google spam policy, I am in huge favor of because it is actually bad user experience. It is a dark usability pattern that some publishers, some websites have embraced as a way to artificially inflate the time on site, artificially inflate pages per visit. and has no benefit whatsoever for the user. And it's only there purely for publishers to make more money. And I think Google with its power it has over the web can make the web a better place. And it has done so before by, for example, pretty much forcing everybody to go to HTTPS, secure connections instead of HTTP. And this is one of those things where Google is using its power to actually make the web a better place. So for this one, I'm all in favor of it. Having said that, You also sort of have to question the power that Google then has over the web, that they can make these policies and basically force the entire web to adopt their rules. Because if you don't, you lose your Google traffic. And that's situation very few websites are comfortable being in. So while I like the policy, it also highlights the fact that Google probably has way too much power over the web as a whole. But you know, that's not something we want to talk too openly about, lest we bite the hand that feeds, right?
speaker-0: Yeah, which is a great song by ⁓ Trent Reznor, Nine Inch Nails. But yeah, right. Loads of SEOs and loads of publishers came out on kind of like, well, maybe not publishers, right? Came out celebrating this because, you know, just reading sort of like some of the comments around when this was announced. was, you know, some publishers argued that... ⁓ You've got to make money somehow, right? And so it's by advertising methods like this or, ⁓ know, audience, inverted commas, audience retention methods like this, you know, it pays the bread and butter for the real journalism to happen, right? But ⁓ I can understand somebody who might be in the commercial department arguing from that. But, you know, I think it's our job as SEOs or as editors, writers, is that you should always be defending the audience experience, right? Because if that's not happening, then who is?
speaker-1: I have very strong opinions on this and I have had arguments with people about this. People say, we have to do this because we have to pay for the journalism somehow. Whereas there are some websites that have gone so far with monetizing their audience that the website isn't a journalistic outlet anymore, it's an advertising vehicle. And yes, I realize we need to make money. We need to fund journalism in one way or the other. slapping these sorts of hyper-aggressive ads on it. And I include stuff that has this built in like a tabula, outbrain, web content. They have features in their software platforms that enable things like back button high checking as well. I do not agree that those platforms offer any sort of positive user experience and that if websites are... content or even happy to host these sorts of platforms and go hyper-aggressive with monetizing every single visit, it shows an inherent contempt for your audience, for your readers. It shows that you don't actually care about the usability, the experience that you provide to your audience, but that you want just to extract maximum value from your audience, no matter what the cost is to that audience, to those readers. And we have to find a bit of a balance there. Yes, we do need ads. We do need monetization. We do need to find ways to pay for the journalism. But there are things that you do that go too far. And websites that go too far are websites that, in my experience, tend to lose in the long run because people start building these subconscious associations that that website is not fun to use, not pleasant to use, and people will try to avoid going to those websites. Whereas I think if you're still focused on providing an optimal user experience within the context of what you can do with monetization, you tend to fare better in the long run. You find it easier to build audience loyalty. You find it easier to maybe to get people to subscribe, to ⁓ pay for your content, to register for registration rules, all those other things that you can do to make loyal audiences. So I think publishers need to be very aware and push back on their own commercial departments. that they don't go too far with injecting these nasty dark usability patterns that serve no purpose at all except to maximize money and decrease the actual value of your website to your actual audience, the actual readers that consume your content.
speaker-0: Yeah, I guess it's, it's really down to if audience teams or, or data teams or UX, you know, product teams, if they're empowered and have got an equal, ⁓ know, equal level with the commercial team. Cause you know, sometimes it may just be that the commercial teams, they've got like the, primary communication channel with the powers that be. And, you know, you really need to be listening to the other specialists in the room that really. No, the audience,
speaker-1: It's a leadership challenge. Leadership needs to set these boundaries and needs to build that internal company culture that makes sure that everything that the company does to improve and to grow doesn't go too far in building bad experiences for their audience.
speaker-0: Okay, agreed. So yeah, so really there's been a couple of reports particularly in the New York Times and saying which Lily Ray is also commenting on around AIO ⁓ accuracy or lack of accuracy. Do you want to talk a bit about that?
speaker-1: Yeah, there was a New York Times article where they did a fairly small study that showed that about one in nine, one in 10 AI overviews was inaccurate to various different degrees. And they wrote an interesting piece about that. Google fired back and said that these were not representative searches part of the study and they were deliberately crafted to elicit bad answers, inaccurate answers. And Lily Way responded to that. Google response saying, actually no, these are fairly representative. This is a normal user behavior or fairly reflective of normal user behavior. And the accuracy rate in AI overviews is about 90%, which means that yeah, one in 10 AI overviews will have a bad response in it somewhere, which shouldn't really surprise anybody by the very nature of LLMs. The hallucination problem is not solved, can never be solved, it's inherent in the technology. So of course, AI overview is going to be inaccurate. But Google wants to safeguard its image. And personally, I don't think the media make enough of the fact that AI overviews are inaccurate. They don't show enough examples. And people just take these AI generated answers as gospel and make very important decisions sometimes. based on the information provided to them by an AI overview or by a chat GPT answer or by a perplexity answer without necessarily double checking it. So I think we probably need to give this more attention and tell people, okay, it's very useful. It's very handy getting an answer from an AI, but at the same time, double check it. Yeah. Especially if you want to make any important decisions based on it regarding your finance, your health, your relationships, and don't accept it at face value ever.
speaker-0: Right. I mean, it's all about regulation, right? mean, it's like using a food example, a fast food example, right? It's like to put clearly label. it well, maybe not food, maybe cigarettes, right? It's kind of like, that took many years to put that kind of warning on it. But it feels, I mean, we're sitting now looking at how the search results look like, which we're familiar with for many years, but... Across all these services there should be some kind of warning, Or treat with caution.
speaker-1: Yeah, a health warning. I also see the other side of the coin here where if you go to a social media website, ⁓ TikTok, Instagram, Reddit, YouTube, and you see something said there by a human, you tend also not to necessarily take that at face value, at least you shouldn't. The difference is of course that we have been trained by the last 25 years of Google to sort of trust their... responses, that there's a certain level of authority built into what Google shows you. with AI especially, with the confidence that AI presents its answers, it tends to really throw people's bullshit detectors off and they tend to just say, ⁓ it's set with such confidence and it comes from Google, therefore it must be true. And I think that's something that we need to maybe teach people how to handle better. that okay, even if it's presented with confidence by a trusted brand, does it necessarily mean it's actually true? And you might want to dig a little bit deeper before you do anything with that information.
speaker-0: Critical learning, critical skills, we all need to sharpen up on them, right? So, I mean, that brings us directly into AI, right? Into the main course, if you like. So, you know, it's been a huge learning curve for myself and I assume you, as you've said, Barry, you there's something new to learn every day, but just from the top of my mind, I saw a post recently which didn't really comment on the latest report or the latest findings. was just asking a selection of 10 to 15 experts ⁓ how they are coping with all the reports which are coming out. Because even if you thought like, I remember when I first started, I was studying SEO, studying ⁓ Google. You know, there is no, I mean, obviously you can read Google documentation, right? But there is no central resource where you can learn everything. It's a much more complex thing. You've got to, you've got to read the latest studies, you've got to network a lot. You've got to kind of like, you've got to test, see, you know, Google might say something, you've got to put it into play and see what actually happens. There's a, it's a holistic approach to it, right? And now you've got ⁓ AI visibility and someone is telling you every hour, every minute on LinkedIn. you need to do, right? So that's what we're here to kind of like, kind of give you some guidance on that. And with that, I'll hand over to you, ⁓
speaker-1: I think there's two things you want to cover today, which is ⁓ AI crawlers, AI bots and scrapers, and how publishers should maybe interact with those, and then talk about whether optimizing for AI is a fool's errand or maybe worth doing in the first place. I want to start with crawling and how AI crawlers are currently behaving, because that's also a bit of hot topic at the moment. is also an accelerating topic because the AI bots are getting much more aggressive. I've seen some interesting data, interesting reports, as Perlman just said, because this stuff just keeps flowing out at like one report a day and it's hard to stay on top of. But this was an interesting one from Tolbit, which has a state of the bots report that they do every quarter. And for the last quarter of 2025, they had an interesting report that showed that There's very aggressive crawling happening by AI bots, primarily by GPT. And most of that is not actually training data. There's two types of crawling. Generally that happens, increasingly three types, but let's talk about the big two types, which is a calling for training data, which is how AI tools gather content to either train an entire model or fine tune and improve an existing model. And then there's ⁓ retrieval augmented generation, RAG. which is when someone actually does a prompt on an AI system like ChatGPT and ChatGPT goes in and basically searches the web for relevant pages and then retrieves them and summarizes them and provides them with an answer and shows them as citations. The training data crawling is less than one-tenth of the retrieval augmented generation crawling, whereas it used to be the other way around. But that also has to do, of course, with how many more users have now adopted ChatGPT and other AI bots. So there's much more usage. So there's less need for AI tools, LLMs to train their models. They already have the data to a large extent and just keeping those training data up to date isn't that intensive a process anymore. Whereas the sheer use case of AI tools means that there's tens of thousands, hundreds of thousands of prompts happening every minute. And therefore there's a lot of retrieval, augmented generation, calls being done. And then there's a third layer of calls, which is the AI agents, but it's still a very early stage. It's not a huge activity there for now. So we don't really have to talk about that too much. What I find really interesting when I look at those sorts of reports and the amount of calling that's happening in that it tends to outpace the calling of classic bots like Google bot and Bing bot. And more importantly, That call behavior from especially chat GPT does not seem to align to the value that the caller generates for the website. As in yes, okay, you get called 150,000 times by chat GPT and you get 12 visitors. Whereas if you get called 150,000 times by Google bot, you get a couple of thousand visitors. Usually the value exchange is very different, which makes it very worthwhile for websites to allow Google to call them and index them and use their content. Whereas the value exchange for ChachiPT is less clear. It merely means what the goal is of you being shown in a ChachiPT generated answer. We can talk about that a little bit later as well. And you have methods of blocking certain crawlers on your website, but those methods are not necessarily foolproof because there's certain workarounds that the AI tools use. Again, to stay with ChachiPT for a moment, they have The main retrieval augmented generation caller is ChatGPT user, which is what they call user-instantiated, which means they only trigger a call when a user asks for a specific prompt. And that also means that ChatGPT feels it shouldn't necessarily respect your robots.txt disallow rules. Because when you browse web pages in a browser, you also don't respect robots.txt. If there's a link to a page, you can just visit it in the browser. whether or not a bot like Googlebot has access to it. And ChatGPT extends that to its ChatGPT user crawls as well, because the user instantiated, they basically see it as sort of a browsing session, and therefore should be able to load the webpage. Of course, there isn't actually a browsing session. The page doesn't actually load. It's just ChatGPT downloading the HTML, extracting the content, regenerating it in an answer and serving it up to their users. So again, there's no value exchange. You don't get a visitor to your webpage, whereas in a browser, of course, you do. So I think that's a little bit of a sneaky method. ⁓ But you can still probably block chat you with a user if you want on a hosting level. But then there's an extra layer, really, really problematic layer, I think, is that there's a lot of third party crawlers, like a lot of them, that will basically scrape the web, everything they can find. And using user agents that you probably have never heard of. And they probably change user agents all the time as well to mask themselves and become different crawlers in your log files, but basically all the same crawlers. And they gather all the data and they then sell it to AI companies. And there's a lot of example of that. There's a few posts on LinkedIn I saw about this as well.
speaker-0: That's a key thing, right? You may have some established AR companies who say we don't do this, but then there's the acquisition of that data. did they get the data from?
speaker-1: The AI company says, yeah, you can block us in robots.txt and we won't call you. Great. But then they still get your data by just buying it from other crawlers, from third parties. So you're not really blocking anything. again, the crawling behavior is very aggressive, tends to ignore robots.txt also because most robots.txt files will not have specific block rules for those user agents because they tend to change user agents all the time or they might not even know that they exist. And some of them just plainly ignore your OSTT file anyway, because they don't care. There's no legal repercussion. They just ignore your block, your digital rules and call you anyway.
speaker-0: Yeah. And I found, you know, some clients have come to me, they're quite confused about what they should be doing with the robots TXT file. Right. So, you know, it's, you know, it's a, it's not a directive. And as you say, there's many ways that the bots can bypass you. But I still think it's a useful thing for publishers to do, you know, for those that do not know, please go and check out the robots TXT file. So it's literally domain, forward slash robots dot txt. ⁓ many publishers, particularly who might be on WordPress or other kind of out of the box, ⁓ CMS solutions may never in their lives have to look at a file like that. But, ⁓ you know, that's where you, that's why I think it's a really good, ⁓ discussion to, ⁓ for publishers who may not have, who may not have explored what they should be doing. with AI bots is to start somewhere and starting is looking at your own robots.txt file or go to a publisher like the New York Times, a well-established publisher, look what they've done and try to explore why they've taken those decisions. That doesn't mean that you copy and paste because there's the key decision-making beside each disallow entry in there. But I found it a useful discussion when there's so many options. Well, let's just start with this. then it's ⁓ then by discussing what's the difference in the bots, which are training bots and which are the ones that we want to keep to our site like Googlebot, right? We don't want to disappear from the SERPs. treat with caution again.
speaker-1: And I've always been a fan of just allowing the bots to crawl and then disallowing the ones that don't behave or that you don't feel have a positive value exchange. But now I actually think we probably need to flip it around because bot behavior is so aggressive. think more than half of the web traffic is now just crawlers and bots. And the bots are so aggressive and basically devious to a large extent that I think we now just need to block them all by default. And yes, do that in robots.txt and then... have allow rules for the bots you do want to crawl. Google bot, Bing bot, some specific other bots that you think there's an added value there. And then just block everything else. And that will actually stop most of the bots. Because most bots too tend to be polite crawlers in the sense that they will adhere to robots.txt dissallow rules. But there's also plenty of bots who just ignore it and crawl anyway. Which is where hosting level blocking comes in, which is not necessarily Well, we're still TXT, but your server, your hosting environment, your CDN detecting bot crawls and saying, you know what, your bottom can block you unless you're specifically allowed. And CDNs like CloudFlare already have these things enabled. And I think more CDN providers will follow. And again, it's one of those symptoms of how the web has changed from an open free access ecosystem to increasingly walled gardens. This is all another extension of that where, you know, and I sound maybe a bit old fashioned or slightly depressed of this, because I remember these old days of the web, Steve, like you do, where it was just a free fall. was like the mild west. could do anything. Launch it and it was good. It was happy. ⁓ If a little bit of a cowboy environment as well. Whereas now everything is going behind walled gardens, paywalls, logins, you name it. And even... you know, access to content by machine systems is now being blocked very aggressively on the hosting level because we have to, because it's self-defense, pure protection, because the economics of the web have now been skewed in such a way that giving away your content for free without any safeguards is Not commercially viable in any way, or form. Unless you're happy to give it away. And there's plenty of people who perfectly happy with giving it away. But if your business revolves around the production and distribution of content, you need to have safeguards in place. And hosting level blocking is the next step. After payrolls, after registration rules, all those things, we now also have to block bots in a very direct, aggressive way on a hosting level. And only I feel allow... bots to call your website where you know you have at least a good chance of a positive value exchange. And that means search engines like Googlebot, and we can complain about Google's diminishing traffic to the web all we want. It is still by far the largest traffic source to the web with no real competition. So yes, you should let Google call your content and index it to show it in search results, to show it in the discover feed, things like that. Some other smaller search engines being bought, let's go and do that Yahoo, whatever other qualifiers are, DuckDuckGo, Baidu if you're in China, great. And then when it comes to the AI systems, you need to have that internal debate whether you want to be part of the training data and be shown as a cited source in their answers, which retrieval of many generation, whether you just want one or the two, no training data, but we do want to be cited as a source, or if you want none of it, and then block all the AI bots by default.
speaker-0: Right. mean, just on that point about being in training data, right? I mean, that's not a quick fix, right? So, ⁓ you know, so if you're in the training, if you're in training data, you've almost got like a pole position, right? Or visibility in there.
speaker-1: Not necessarily. The training data is useful to a certain extent, but because nearly all the prompts now also rely so heavily on retrieval augmented generation, the training data is basically just there to determine the quality of the model in the first place. How good the LLM model is in terms of the quality of its output, the accuracy of its output. I'm actually more okay with being part of the training data than part of the retrieval augmented generation nowadays, because the training data, just because the LLM model is out there, the technology is there. And we want these tools to be as good as they possibly can be, but not putting Pandora back in its box. okay, let them be trained on my data. I think being part of the training data also means that you can build associations, that if you have a strong brand in a specific area, for example, SEO, and you're mentioned in the context of SEO very often, I think in that initial ⁓ generated answer that is not augmented, that is purely constantly LLM, there's a higher chance of you being cited in it because that association already exists. Then the retrieval of the generation where the call or ⁓ the LLM actually goes and has a generated answer, let's see how accurate it is, or some LLMs do it the other way around. They start by retrieving and then summarize it. That's a different sort of value exchange. Then you have to decide, right, it's a lot of call activity on my website. There's a lot of load being placed on my server. by these retrieval systems to get me a mention in an answer in some context or another. And then you have to decide, is that worth it? Can I attach a value to being part of that answer that has a positive economic value for me somewhere down the line? And that's a much more difficult question to answer at the moment.
speaker-0: Yeah. And you know, it's really interesting what you say because I was reading an article in Digiday by Sara Gialoni, who was writing a response piece to this information out there, that third party bots are scraping a website site mad. She had this great quote, which I've got here, which is, said, most brands have no idea whether they're being mentioned or ignored by Chatjibi Tea, Gemini and Perplexity when users ask questions in their category. There's no rank tracker. for AI generated answers, no impression data, no notification when a competitor gets recommended instead of you. The result is that most content teams are investing in AEO based on theory, not measurement. They're publishing content, optimizing for AI search and hoping it's working with no reliable way to know whether it is.
speaker-1: Yes. I mean, there's a lot of talk about prompt tracking and its accuracy and the truth is prompt tracking is just not accurate in any way, or form. There are some studies on this actually that show that you can do the exact same prompt 10 different times in the same AI bot and you get 10 different answers, 10 different set of citations, 10 different brands mentioned. Now there are some consistency signals there for some brands, especially brands that have very strong positions in their market. And therefore they will always be mentioned in one way, shape or form. as part of a particular answer. But prompt tracking in itself, I feel is such a colossal waste of time and energy because it doesn't really give you meaningful data. In fact, it gives you bad data and bad data, in my opinion, is worse than no data at all. So I think, you know, if you're currently paying a lot of money for prompt tracking and whatever shape that that takes for you, I recommend stop that. Don't do that. Maybe do some experimental prompts now and again if you want to show some data to some senior level executives in your company. But as a whole, the exercise is fairly pointless because of the inherent instability of an AI-generated answer.
speaker-0: Right. right. So, mean, other options, we will go into a list of things that you can do from the skills that we have between us. But I would always call out, you know, ⁓ obviously having a, ⁓ having a view, a strategy, an AI bot strategy is obviously a key thing for publishers to do. ⁓ but it's not the only thing they can do. And so I would always argue that they should, wherever possible. publishers in general should join part of their collective bodies to have more of a clout. Now, ⁓ there are many options out there. We've got the PPA, the AOC, you've got the Newspur Collective with the likes of the Guardian, Financial Times, etc. All these are great steps forward. ⁓ But I wanted to recommend here a Ricky Sutton's podcast, Future Media. where he was ⁓ speaking to an expert that was previously a Danish MP who is part of a new collective there. It's a really fascinating podcast because it just demonstrates the power of working together as a collective. There are obviously less publishers in Denmark, but they've all got together. and ⁓ they're making a difference, partly because of how the political system is set up, but there just seems to be a moral agreement among them that we can't have this misinformation ⁓ and our business is going to go if we don't do something. So we need to stand up together as a collective to do that. So I think we can all take a lesson from that. And that was a great listen to.
speaker-1: Absolutely. The classic tactic of divide and conquer is too powerful. And the only way the industry as a whole can get meaningful change done is by combining forces, which is one of the reasons why I like this pro-coalition here in the UK so much. Where they have such massive media brands associated with it. And I think that'll be absolutely key to get government regulation in place to protect intellectual property, to protect copyright. and protect the sanctity of journalism in this country.
speaker-0: Right. So, do you know, one of the things I wanted to pick up on is everything that you've just said about the low value exchange for publishers, which, you know, we obviously get, particularly in news, because what you click through is the product, right? So if that's not leading to a subscription and per click that it's kind of like low value anyway. ⁓ You know, it just taking, that away from publishers is a real kick in the teeth. But, um, you know, when you kind of look at LinkedIn and other sources of information and you see the big evangelists of, uh, uh, prompt tracking or just AI visibility in general, I often think, well, actually that's because they are connected more to a brand, uh, capacity, right. Or to like, like an automobile, right? So if you're, if you're selling 30 grand. 50 grand, 100 grand cars, then obviously you don't want to lose any position. you know, that's why you're seeing in brands, you're seeing a lot of teams being ⁓ built in house, right? Where there might be more of an argument to battling against competitors in that space. But obviously, you know, if you talk to most news publishers, there doesn't seem to be this kind of... assortment of teams assembling ⁓ to build AI visibility teams, right? It seems to be more something that SEOs or audience teams are kind of looking at. I mean, would you agree just because the value exchange is so low, that's why it's not happening?
speaker-1: Yeah, yeah. If you have a direct sale opportunity or even an indirect sale opportunity and part of a customer journey through a funnel, then being visible in AI-generated answers has value because people do use it to do research. ⁓ Sticking with your car example, you can imagine people asking what is the most reliable car, what is the best value for a money car, what is a ⁓ fairly good affordable luxury car, things like that. And if your brand matches those sorts of queries, you want to be shown in those answers. So optimizing for AI visibility in those contexts, very important. Don't underestimate it. You probably want dedicated teams focusing on that. Even if you can't directly measure it. But that's the same way we have it classic advertising. You you don't know what part of it works, but you know that it works, so you just keep doing it. And the same with AI visibility, you sort of need to see it as indirect advertising in AI. If, however, your output your product is content, you have a very different context when it comes to AI visibility, because the output of AI visibility is content. That's what they do. So if you are part of an AI generated answer as a content website, there is no need for the user to go beyond that. There is no secondary conversion down the line. They just interact with your content through the medium of an AI generated answer. and there is no need for them to visit your website, even if you're cited as a source. If you are a traffic-dependent website, which most publishers are, there is no value, none whatsoever, in being visible in an AI-generated answer, because the traffic just isn't there. Even the highest click-through rate we see on AI-generated answers, which is in Google's AI overviews, It's 30 to 40 % lower than a classic search result. Now that still means that's traffic potential and there probably is still some value exchange being shown in an AI generated answer on Google. But then if you go to the next big boy, which is of course, ChatGPT, it sends almost no traffic to the web. In fact, I think ChatGPT a few months ago even changed its citations and and external links in its answers to keep people on chat GPT more often because they felt too many people were clicking through. The highest percentage of traffic I've seen to a news brand from chat GPT was I think the New York Times, and this is public data so I'm not giving away any secrets because I don't have access to the private data anyway. But I think it's like less than 1 % of the total referral traffic comes from chat GPT. And I think on average, New York Times like 30, 40 % comes from Google sources. So we're talking like an order of magnitude of for variance there. So that makes it a very interesting conundrum for publishers. And then they have to maybe look at what secondary benefits there might be to still be incited as a response in various contexts. You can imagine, for example, a website like the Financial Times, which is a very hard paywall website, that they still probably want to be shown in certain contexts for financially relevant queries in these AI tools. because it also shows them as a trust brand, as in, hey, we have this information, you want to subscribe to us, you want to come to us if you want to directly from the source. And that works if you do have that higher level of brand perception or brand power and a product that you can sell in association with that, which in the Financial Times case is the hard paywall content. But if you're more of a commodity news website and you have fewer distinguishing content types, It's very difficult to justify optimizing for AI because there doesn't seem to be any value in it for you as a publisher to do that.
speaker-0: Yeah. I mean, ⁓ it's only going to be the bigger publishers who are going to say, we're going to invest in this because we want to fight misinformation, right? Because they've got the capacity to do it. There might be other reasons like, mean, ultimately it's, you know, a lot of AI visibilities around brands, reputation, right? Who's citing you, who's quoting you, what the other's saying about you. Okay. That's popping up in AI. We need to, we need to look at that, which you could argue we've done many times before in SEO. So I'm not just not going to that, but, ⁓ you know, I think we want to kind of like move on to kind of like things that we can do from a technical SEO to, help publishers. Right. So we've already had a comment podcast before, even in our first podcast, which was around, ⁓ you know, at the, at the source, there's, ⁓ there's a lot that you can do in technical SEO in order to advance your, your AI visibility. ⁓ but We'll just kind of like go through a couple of them, right? So I think we've got semantic HTML.
speaker-1: Yeah, semantic HTML specifically for retrieval augmented generation, it allows an AI system to very easily identify boilerplate content versus core article content and use that core article content then to be ⁓ used as a citation in the AI generated answers.
speaker-0: Yeah, we've also got, I mean, this is a classic one, subheadings as questions, right? I mean, we've done a lot of that, particularly not in explainers for SEO, but these pieces still, ⁓ they appear a lot. I mean, on that, it's interesting about evergreens, right? Because some publishers have pulled away from doing that, but ⁓ it's a way to demonstrate added value, right? To the users, because you're... ⁓ First of all, you're listening to the audience about what it is that they want to know, and you're putting it in one place, which is added value, sitting alongside the actual news story.
speaker-1: Exactly. I think when you understand the questions that your audience asks and you put those questions in your actual content and then answer them in the subsequent paragraphs, you're helping your users because they just get the direct answer to what they're looking for. And you're helping the LLMs because they know, ⁓ you've got the answer to that particular question that the user has asked. So I'm going to use your content as a citation.
speaker-0: Yeah. And structured data, right? That's been sort of like debated because I mean, this is why everything flip-flops, you know, from day to day on what's the general consensus on this. But I think we got into some position at some point where some practitioners were saying, ⁓ you know, if you just concentrated on unstructured data, that's definitely going to advance your mentions in your citations. And then others were saying, well, wait a minute, what is actually the content on the page, right?
speaker-1: Yeah, the jury is still out on that one. There's one particular context where we know structured data helps and that's agentic optimization. If you want to optimize for AI agents interacting with your content, they tend to look at structured data because it's basically a machine readable summary of your web page and allows them to extract the right information, make the right actions. For pure content-driven AI citations, the added value of structured data is not that clear. But structured data generally has positive effects, especially if it's a schema type that Google supports. So always a fan of implementing it when you can.
speaker-0: Yeah. Okay. So chunking, right? So chunking ⁓ has been, I mean, we're in this interesting environment where we're all learning about LLMs and then anyone that says or identifies a pattern will come out and say, you need to do this. then, you know, the industry is just too young to latch on to anything with any concrete evidence, but we're still there. looking for the main trends, the main patterns, right? But it's interesting about this element of chunking. So looking at paragraphs, ⁓ even that has come under fire. you know, even Google spokespeople have said, you know, it's not a long-term strategy to chunk everything. However, from my understanding, you know, what chunking actually indicates is really that ⁓ you should look at line by line. So really for AI visibility, it's all about just getting to the point. So now this is interesting a point, right? Because you've got to write for users and write for the bots and what do you do? We're always looking for a happy medium between the two, but sometimes I want to read and I want to subscribe to copy which sends me on a journey, right? So often opinion writing or commentary or just humor writing, that is, comedy writing. I'm sounding like a bot now. It's meandering and like I'm talking now, just, whereas for AI bot optimized copy, it's very much line by line ⁓ optimization. Get to the point, put the main facts in, the main entities, move on to the next point. No dilly dallying. that's the kind of approach that you take and you know there is an art in that ⁓ but yeah it may be less so for news publishers.
speaker-1: Yeah, the chunking is always a bit misunderstood, I think, by many SEOs. I don't pretend to understand it fully myself, but I do know that in the context of LLMs, it's about the context window that an LLM has to understand a sentence or a paragraph, which is not as big as people think. It's not a huge context window, which is why what you just said, get to the point, know, short, concise sentences, that is what the LLMs want, because it allows them to understand the point of your sentence in that small context window that they have. but it doesn't make for pleasant reading. It doesn't take you on that journey. Which is why I am a strong fan of not applying this as a direct tactic for human focused journalism, but to write for humans, to craft that narrative, to build that story and just make sure it's pleasant to read and engaging with the article is pleasant for your users. I think that is a much better long-term tactic than trying to artificially... break your content up into short snappy sentences that are just not fun to read.
speaker-0: Yeah. And also, there's also a side note here on using, actually, when you're using AI to help create copy, just be very careful. I think the general consensus there is, you know, it's fine to have AI assisted copy to help you. But if you are using AI to just generate the actual idea of the article, be very careful of that because it's only sourcing what it's already read, right? So will it mention you if you're just redoing something it's already got? So it's definitely worth being careful there. This is an old favourite of mine, crawlability and crawl depth and internal linking, right? Yes. I think that's our speciality, isn't it? Yes.
speaker-1: I think there's a lot of this is actually a lesson that publishers can embrace because not a lot of publishers allow deeper easy access to the article archives. I think that's a missed opportunity in the context of search to a certain extent, but especially in the context of AI, where every article that an AI has access to both for training data and for retrieval of mental generation improves your presence in their systems. And there's too many publishers that just have like a category page with 20, 30 articles and that's it. And maybe there's some pagination if they're well set up to maybe 10 pages of articles, which again is fine for users. But there's very few publishers who allow deeper access to older content with archive links, HTML sitemaps, things like that. And I think that's probably a missed opportunity for many publishers. If you are a high quality publisher and you have a whole archive of tens of thousands, hundreds of thousands of good journalistic articles, you should probably build a pathway. for crawlers to find that in a very easy way through a date-based archive or some sort of HTML sitemap. So that it's only a few clicks away from your homepage. One example I will quote for this for a website that I think does this really, really well is TechRadar. TechRadar has like nine pages of pagination for any one of the tags and categories. And then there's a link to their archive, which is a unified archive, date-based, which basically allows you to find every article published on their website, on the techradar.com website, in its entire history with maybe three clicks. And I find that a very elegant solution. Some people worry that that sort of solution creates a lot of crawl overhead for Googlebot. It really doesn't. Googlebot knows to focus its crawling generally on new pages that it hasn't seen before. And we'll call the old pages once in a while to check that they haven't changed, but usually only when it has crawl effort to spare. So we shouldn't worry too much about that. And I find that sort of a solution a very good way to enable that all the content to be called and indexed and also to be visible to the AI box.
speaker-0: Yeah, it's interesting because when I've looked at particularly news publishers who have visibility in AI and chat, GBT or perplexity and the like, I've often noticed that it's links to the archive. Right. So that, mean, it's an AI bot dream, right? You've got everything there for it and it's easily accessible. So yeah, I've noticed brands like the Independence get great visibility through that. So yes, go and have a look at the archive system ⁓ on publisher sites. Okay, MetaTags and OpenGraph.
speaker-1: Yeah, still used by both search engines and AI bots, know, the title tag, canonical tags, meta descriptions, open graph title, open graph description, open graph images even. Because when a chat GPT shows a citation for a webpage, it generally has a thumbnail image associated with it as well. And that stuff is just classic SEO as well. You want to make sure that they will find in an order and, you know, properly optimized to give meaningful descriptions and titles for your content.
speaker-0: Okay, I agree with that. obviously with titles, particularly around optimization around Google Discover, right, which is still, we know, big.
speaker-1: Open Graph especially. think the last date I saw was that 60 % of Discover headlines come from the Open Graph tag, not the title tag. So that's an interesting one to keep.
speaker-0: Yeah. Okay. LLMs, LLM text.
speaker-1: Yeah, again, a jury is still out. I do see data that some crawlers, especially chat GPT, do tend to call LLMs.txt files with some regularity, but it's not on officially accepted standard yet. and most of the implementations come off the back of just basic default functionality of like an SEO plugin on WordPress. So, publishers might not even be aware that they have an LLM.txt file. If you don't have it enabled at the moment,
speaker-0: Yeah.
speaker-1: I do not necessarily see value in creating one just yet. I think we need to wait and see if it actually becomes a standard for LLMs to use it. At the moment, I don't see that there is a good business case for it yet.
speaker-0: Yeah, mean, like there's even this week there was ⁓ a discussion around the benefits of doing that. But for the last year, there's been a consensus of not really doing it. again, literally things can change within days of in this fast moving world of AI visibility. I think we've covered a lot of topics there.
speaker-1: I want to talk about one more thing because it's a bit of hot topic as well at the moment, which is Markdown. Markdown is basically a stripped down version of the content of your website served in machine readable format, which you can serve hypothetically to AI crawlers so that they don't get your full website, they just get the Markdown version of your website. And that is seen as a way to basically present an AI crawling optimized website to crawlers from AI machines and present new regular users with
speaker-0: Go for it.
speaker-1: normal web pages. But again, it requires development effort, requires investment and resources in that technology. And the benefit to cost ratio isn't really there because LLMs have, since their inception, been very, very good at just calling HTML pages, standard web pages. That's what they're designed to do, to basically separate out the boilerplate content, separate out the HTML tags and look at the actual content of the page. They don't need markdown. They really don't. And what you're doing if you're giving a markdown version of your content to specific crawlers and a standard version to users, you're falling into the same trap we fell into with accelerated mobile pages. If you remember that AMP, you basically have an entirely duplicate version of your website for one specific user context. And that just creates an entirely separate development, parallel development track for your website. That is a ⁓ potential source of all kinds of conflicts and problems. And especially in the context of markdown, there isn't really a value exchange there because the LLMs already are very good at parsing HTML. So I think when it comes to markdown, that's a big no at the moment and really not worth the investment.
speaker-0: Right, steer clear. Okay, right. So that's a lot to consider. I think that was a hearty meal this time. Okay, right. So I think we're going to sign off there. Obviously, ⁓ we would always cherish feedback and comments and recommendations for what to cover on this podcast. We will be attending Brighton SEO in a couple of weeks, won't Barry?
speaker-1: Yes, we are. I think two weeks from now, two weeks and two days. I look forward to it. I tend to only go to the spring edition rather than the autumn edition. So the spring edition is coming up. And for me, Brightnessio is all about the people who are there and the connections that you make and catching up with old industry friends. So I look forward to seeing you there as well, Steve.
speaker-0: Yes, I'll be there with ice cream in hand, although I think last time it rained. But anyway, we'll be down on the beach, our toes in the water, maybe with a beer in hand. But like, please remember to, I'm going to do that thing of please comment, like, subscribe on our podcast, wherever you can find it. We're across all platforms. And like I said, really would look for recommendations to discuss on future episodes. But for now, thank you for listening.
speaker-1: Thank you very much, everybody.
