Are you eager to boost your website's performance on search engines like Google but unsure where to start or what truly makes a difference in SEO?
Then "SEO Is Not That Hard" hosted by Edd Dawson, a seasoned expert with over 20 years of experience in building and successfully ranking websites, is for you.
Edd shares actionable tips, proven strategies, and valuable insights to help you improve your Google rankings and create better websites for your users.
Whether you're a beginner or a seasoned SEO professional, this podcast offers something for everyone. Join us as we simplify SEO and give you the knowledge and skills to achieve your online goals with confidence.
Ask most of the SEO world about llms.txt and you’ll get a short answer: nobody reads it, it won’t help you rank, don’t bother. On the other side, people are selling it as the way to get cited in ChatGPT. Edd Dawson thinks both camps have it wrong.
In this episode, Edd looks at what the log data says about who actually requests llms.txt files, and why the answer isn’t who most people expect. He explains the one job the file does do, why that job is growing, and the single question that tells you whether your site needs one.
If you do need one, Edd covers how to do it properly, because many of the files out there are no use even to the things that read them.
You don’t have to guess whether anything is reading yours. Salience shows you what’s requesting your llms.txt and where it goes next, straight from your server logs, with no tracking script to add. Try it at salience.com.
Two slips in the recording you may want to fix or note before publishing:
Grosser’s title: you call her Vercel’s CEO. She is the COO; the CEO is Guillermo Rauch.
The 300,000-domain study: the transcript has “Search engine rankings”. The study is by SE Ranking, so check whether that’s a transcription error or what you said.
SEO Is Not That Hard is hosted by Edd Dawson and brought to you by KeywordsPeopleUse.com
To get a personal no-obligation demo of how KeywordsPeopleUse could help you boost your SEO and get a 7 day FREE trial of our Standard Plan book a demo with me now
Hello, and welcome back to SEO Is Not That Hard. It's me here, Ed Dawson, as always, and today we're talking about llms.txt. Now, if you ask most people in the SEO, SEO world about llms.txt, you'll get a very short answer. "It's a waste of time. Nobody reads it. It won't help you rank. Don't bother." And on the other side, you've got people selling it as a thing that gets you cited in ChatGPT. Now, I think both of those are wrong. The first group are mostly right, but they stop one step too early. The second group, they're now selling you something the data doesn't support. So here's where I stand, okay? Llms.txt does next to nothing for search, and I'm gonna show you the numbers on that. But there is one job it does do, and that job is getting bigger every month.
Speaker 2
If you know what that job is, you'll know in about 30 seconds whether you need one. So today, what is it? Where it's use- useless, where it's useful, and how to tell which side of that line you're on.
So let's start with what llms.txt actually is, because a lot of the argument comes from people talking about different things. It's a plain text file, it's written in Markdown, and it sits at the root of your site, so yourdomain.com/llms.txt. And it was proposed back in September 2024 by a guy called Jeremy Howard. Inside it, you've got a title, a short summary of what the site is, and a list of links to your most important pages, each with a line describing what's there. Now, ideally, those links point to clean Markdown versions of those pages, not the full HTML. There's also a companion file called llms-full.txt, which is everything all in one big document. Now, the important word in all that is proposed. This isn't a standard. There's no standards body signed off on this, and none of the big AI companies have actually committed to reading it. Now, if you compare that to robots.txt, which every serious crawler checks before it touches your site, um, you know, that is something that has been agreed upon and there are standards around it. So two things it's not. It's not robots.txt for AI, it doesn't control who can access what, and it's not a ranking signal. Nothing reads it and decides to trust you more. Think of it more of like a contents page. It's like a contents page which is only useful if somebody picks up the book to read it. You know, a contents page without a book or without anyone reading it is pointless.
So let's talk about, yeah, where it's not useful. Let's deal with the skeptics first because they're right in a few places. So for Google, for example, Google has been clear about this. In May this year, um, in their very first official guide to AI search, they listed llms.txt among the things you don't need for AI overviews or for AI mode. Then in June, they added a note saying Google Search doesn't use these files at all. Having one won't hurt you, it says, but it won't help you either. So if anyone tells you llms.txt will help you rank on Google or get into AI overviews, that's just nonsense, so ignore that. Next, on AI citations. What about ChatGPT, Perplexity, you know, all the rest? Search engine rankings looked at around three hundred thousand domains and about one in ten they say had an llms.txt file, and they found no clear connection between having one and being cited more often by AI. So that l- brings us to say does anything ever even read it?
So this is the part that's most telling because it's log data, and you know I'm big on logs right now. Um, Ahrefs looked at about thirty-eight thousand sites that had a valid llms.txt and, and that was back in May. Uh, ninety-seven percent of those files got zero requests as far as they could see. Not a few, zero. No bots, no humans, nothing. Only around eleven hundred got any traffic at all on llms.txt. Other studies showed similar things. One tracked, uh, more than half a billion bot visits, AI bot visits over ninety days, and just four- four hundred and eight of those were apparently requests for llms.txt. And then it gets actually quite, quite, quite ridiculous. It gets funny because when Ahrefs looked at who was making the request that did happen, about a fifth were SEO audit tools and another twelve percent were tools built to scan and study llms.txt files. So like a whole big chunk of this was, uh, of traffic to llms.txt files was the SEO industry checking whether people have them and probably lots of them recommending that you should have one for search. Um, apparently Slack bot fetched these files more often than Perplexity bot did.
Speaker 4
So who does actually fetch it and use it? So in that same Ahrefs data, AI bots of all kinds, uh, made up about 19% of the requests. And when you split that 19% up, it's, it's not what the llms.txt salespeople who are trying to sell it to you for search purposes would expect. The bots tied to live AI search, the ones who would actually cite you, were about 1%. The biggest group was coding agents at 10%, and the, and the top named bots fetching those files was Claude Code.
So think about what this means. It means that the th- the thing reading an llms.txt is not a search engine deciding whether to send someone to you. It's an agent in the middle of a job trying to work out how to use something. These are two completely different moments. In search, the question is: Which page best answers this person? An llms.txt doesn't help with that. With an agent, the question is: I've been told to use this product, so how does it work and where's the documentation? And for that question, a clean contents page with links to clean markdown is exactly what you'd want. And it's worth being clear about how agents choose a product in the first place because it isn't on this file, okay? One study looked at more than five thousand coding agent sessions. Claude Code only searched to web about thirty percent of the time. The rest of the time, it went with what it already knew from its training or the context of the conversation it had already been having with somebody. Codex, on the other hand, searches nine- ninety-four percent of the time. So that shortlist comes from training data and from web search. So that's brand, content, and SEO. That's the stuff we talk about every week. Now, Ahrefs found something else on top of all this, and that's on sites that had no llms.txt, AI bots never asked for one, not once. They don't go looking for it. They read it when something points them at it. So llms.txt isn't a discoverability tool, and it never was. It's the index to your instruction manual, and it only matters once an agent is already trying to use you. So where is it actually useful?
You know, where, where should we actually apply it? Well, let's look at the three places I think we should be looking.
Speaker 5
And there's three places: documentation, APIs, and developer tools So this is the clearest, you know, the clearest ones
Speaker 6
A developer is working in Cursor or CloudCode, and they say, "Add payments to this app or deploy this for me." The agent now has to go and work out how your product works. It needs the right documentation page quickly and in a clean format without wading through navigation menus and JavaScript. Every page it reads costs time and tokens. And llms.txt gives it the map. It says, "Here are the docs. Here's what each page covers. Here's the markdown version." That's why coding agents are the ones turning up in the logs. And you can see in who's adopted the file. One scan of the top 100,000 domains found about 39% of ai.ai domains and 25% of the io domains had one. For .com, it was only 13%. The people publishing llms.txt are the people whose customers are already using agents. And this matches what we see in our own data at Salient. On most sites, llms.txt, if it's there, just sits there. But on sites built for agents where the product itself can be used by an agent, the file gets fetched, and it gets fetched at the point where the agent's working out how to use the product. The best public example that I've come across is Vercel. Their CEO, uh, Jean de Wittig Groser, she was on, um, Marketing Ag- Marketing Against the Grain recently, and she said that on January this year, 3% of deployments in Vercel were triggered by an agent. Ten months later, it's over 60%. Now, I want to be careful here because she did not say llms.txt caused that, but she put it down to years of open source work, a huge amount of documentation and a, and a big developer community. Agents learned about Vercel from all of that. But she also said that every time they ship content now, they ship a markdown version in llms.txt alongside the human version as standard, and every product feature has to work for an agent as well as for a person before it gets shipped. So for Vercel, llm- llms.txt isn't a growth hack. It's one small part of treating agents as users, and we'll, we'll, we'll talk more about this in future podcasts. But when most of the activity on your platform is coming from agents, writing your documentation for agents stops being something that's optional Now the last one, now this is, you know, this is a, a thing beyond developer tools, but this last one is early and, and but it's where it's heading, I think. And that's Vercel's own online shop template uses llms.txt to tell an agent how to buy something on a shopper's behalf. So not just where the pages are, but how to complete the purchase. But I'm not saying every shop needs that today, but as more of the web gets used by agents doing tasks for people, the more sites will have something an agent needs to be told how to use.
So should you bother with llms.txt yourself? I mean, how do you know which side of the line you're on here? So the, the question is simply, would an AI agent ever need to use my product or read my documentation to finish a task for someone? If you ask that of yourself and of your product and of your website, if the answer is yes, that's developer tools and APIs. It's software with integrations. It's anything with a big help center or a knowledge base where you want the agent landing on the current page and not the one from three years ago. And increasingly it's shops, if you want agents to be able to buy from you. If the answer is no, though, if you, if, if you don't think that is the case, if you're like a-- if you're a local plumber or you run a blog or you've got a brochure site, nothing's gonna come looking for that file. It won't hurt you. It just won't do anything. Spend that hour instead of, you know, checking you're not accidentally blocking AI crawlers instead, if you wanna spend some time doing something useful. And if your platform makes one for you automatically, which Shopify and some of the SEO plugins now do, leave it alone. You know, it's free. Just don't expect much from it But if you are in that group that says yes, it's worth doing it properly.
So there's a few things, because a lot of the files out there are useless even to the agents that read them. First of all, make sure the links in there, they're the whole point. That same scan of the top one hundred thousand domains found about one in six llms.txt files didn't list anything for an agent to go and fetch. That's a contents page with nothing being pointed at. Secondly, describe each link in a line. So the agent's deciding which page to open. You need to give it enough information in advance to help it choose Thirdly, point to clean markdown versions of the pages where you can. That's where the saving is. That's where the, um, agent itself will be able to understand it more swiftly than if it's got to pass through HTML first. And fourthly, organize it around what someone is trying to do, not around kind of your internal departmental structure or anything like that. Try and make it task-oriented. And fifth, tell agents where it is so they don't go hunting for this file. So link to it from your docs, from your quick start, reference it in any skill or rule files you publish for coding agents. And, and finally, keep it up to date. A map that sends an agent to a dead page or an old version is worse than no map at all.
And here's the thing that strikes me about this whole debate. People have spent two years arguing about whether anything reads llms.txt, and you don't have to argue. You can just look. Every request for that file is in your server logs. Who asked for it, when, and what they asked for next. And that's what we built Salience for. You know, at Salience, we read the server and CDN logs for your site it's already producing with no track-tracking scripts to add. So you can see whether anything is requesting your llms.txt at all, which agents they are, whether they're real, and which pages they're gonna go and read afterwards. If you'd have-- If you've had an llms.txt up for months and, and nothing has touched it, you've got your answer. You know, noth- you don't need it. Nothing's looking for it. And if coding agents are hitting it every day, you've got a different answer. You know there's a good reason there to make that file better because you know people are using it.
So if you want to try it out for yourself, link's in the description or just go to salience.com. So wrapping up, llms.txt is, it's not SEO. Never was. It won't help you rank, and it won't get you cited. On that, you know, the skeptics are actually right, but it's not nothing. You know, w- in, in this world that we're working in now with AI, when those AI agents are going searching for your documentation, then this is a file that is useful for them. If they can find it, they c- they will use it. They're using it to search around your site to find out what to do. So maybe it is SEO after all. Who knows? But you just need to know which of two problems you've got, whether the agents are coming to your site and using you or whether they're not, and then you know whether you need one or not. So I hope that's been useful. Thanks for listening, and I'll see you in the next one.