Speaker 1: 0:54

Hello and welcome to. Seo is not that hard. I'm your host, ed Dawson, the founder of the SEO intelligence platform, keywordfupoleasercom, where we help you discover the questions people ask online and learn how to optimize your content to build traffic and authority. I've been in SEO and online marketing for over 20 years and I'm here to share the wealth of knowledge, hints and tips I've amassed over that time. Hello and welcome.

Speaker 1: 1:15

Back to another episode of SEO is not that hard. It's me here, ed Dawson, as usual, and today I'm going to be talking about how you can find out how and when Google is crawling your site and if it's finding issues with that crawl, all those kind of things. Now, obviously, it's important when we're considering our website and how Google is crawling it. The crawl phase is when Google's going to discover changes to our pages and discover new pages, find pages that have disappeared, all those kind of things. This is the phase where Google is visiting your site and downloading the content on it. And being able to understand how Google is doing these crawls, when these crawls are happening, if there are any problems with these crawls. It's really, really important. So we're going to talk about why it matters and how you can use it to improve your site's performance. So what is the method for finding this? Well, we look at the crawl stats report in Google Search Console. If you're not familiar with Google Search Console, go back. I've got other episodes that talk about Google Search Console. If you're not familiar with Google Search Console, go back.

Speaker 1: 2:17

I've got other episodes that talk about Google Search Console. It's a way where you can verify that you own a site with Google and then Google will give you information with extra information on how it sees your site, how your site appears in search and all those kind of things. But now this Crawl Stats Report is a section in Google Search Console that provides detailed insights into how google crawls your site. So it's a bit like a health check for your website's crawlability, and the report will show you total crawl requests, which is how many times google pot has visited your pages. The crawl response time, which is how long your server is taking to respond. Download size, the volume of data is fetched during each crawl session. And crawl errors, so issues like 404 errors and other server errors that may be confronting proper indexing. Now these metrics can help you understand whether Google is efficiently crawling your site or if there are issues that might affect your SEO.

Speaker 1: 3:10

So how do we access the crawl stats report? Well, it's relatively straightforward. All you need to do is log into your google search console account, select your website property and then navigate down the list to the settings um link, which is one of the little cog next to it, um, so it's kind of buried. So I think a lot of people miss out that this is actually available to them. So you click the settings link and then you will see crawling information. So there is the crawl stats one there and it tells you how many crawl requests have been done in the last 90 days, and then you can click the open report button to open up the report and it will give you a graphical interface that then gives you a whole list of that you can dive down into, and these visuals make it really easy to spot trends like a sudden spike in crawl requests or an unexpected drop in crawl activity.

Speaker 1: 4:03

So let's have a look at what each of the metrics are that you're going to find in this stats report. So first of all is the total crawl requests, and this number shows how many times Googlebot has crawled your site over a given period. A steady or increasing number can indicate healthy site activity. However, a sudden drop might signal an issue like a server problem, access problem, server downtime, that kind of thing. The secondly, we've got average crawl response time. Now this metric tells you how long it takes your server to respond to googlebot's request. So a long response time can hurt both user experience and your se SEO, and it could be due to server overload or other performance issues you might need to address. If Google discovers your site is suddenly slowing down when it's crawling, it will actually back off its crawl so as not to cause your server issues. So you always wanna make sure your server's fast, up to date, you know can respond to requests, because you don't want Google thinking your site is easily overwhelmed. Thirdly, we've got total download size. So this is the kind of volume of data that Googlebot is downloading during its session Monitoring. This can alert you to unusually large downloads, perhaps due to heavy images or media files. You know that kind of thing, the kind of thing that might slow your site down.

Speaker 1: 5:11

And then you can see crawl errors in these reports as well, which highlight any errors encountered during crawling, such as 404 not any sort of not found pages aren't found, or or 500 errors, server errors. These errors can negatively impact your site's indexing. They're not resolved promptly. Understanding these metrics it gives you a clear picture of how your site is being crawled and it helps you spot areas that need improvement. So why is this data important? Well, the Crawl Stats report can help you optimise your site in several ways. So it gives you a way of sort of proactively detecting issues that are occurring. So if you regularly review this report, you can catch errors before they become major problems. So, for instance, if you notice an uptick in crawl errors, then it might be time to investigate server issues or ready-broken links.

Speaker 1: 5:56

And with all these crawl errors, you can click into them and it will give you examples of the page giving you those Crawl budget management. Now you don't have a budget that you can decide where Google will go, but the crawl budget is effectively the number of pages that Googlebot will crawl during each visit, and it varies by site. By monitoring your crawl activity, you can ensure that Google spends time on your high priority pages rather than getting bogged down with duplicate or low value content. Now, this is a problem that's only really going to be on larger sites. If you've got a smaller site, then you know you're not going to have an issue with crawl budget. But if you've got thousands, tens of thousands of pages, then you know there is going to or even more, than Google is going to only decide to crawl so many at a time because it just doesn't have the ability to crawl the entire internet every day from beginning to end. It has to eek that out amongst the site. So if you've got a very large site, you might want to be checking that you know you're directing google into the right places.

Speaker 1: 6:55

Server performance insights An increase in crawl response time might indicate your server is struggling. So this is like your cue to go and look into performance improvements such as better hosting, caching solutions like a content delivery network or code optimization. If you've got particularly slow bits of code on your site and then you're looking out for content updates and trends, so you might notice spikes in crawl activity following new content updates or major site changes. If you've implemented anything like that and this feedback loop helps you understand how quickly google can recognize your updates and can give you a guide on the best times to schedule kind of new content launches so you don't overwhelm. So what are some of the things that you need to be thinking about with this crawl stats. What's the best practice for using this information? Well, first of all, you want to monitor regularly. So you know, set yourself a schedule, look at it weekly or monthly, to check your stats report. Put it into your kind of the cadence of your rhythm of the stats that you look at and regularly monitoring. It helps you spot any trends or anomalies early. If you're doing major site changes, whenever you're all like new content or design updates, check your crawl stats before and after the changes.

Speaker 1: 8:09

Look for unexpected shifts in this quality of two errors that you might have caused in the crawl. If the things you might. For example, you might see a spike in 404s. I mean you need to need to put 301s in, for example. You know, fix these errors promptly. You know, don't ignore 404 or server errors. These issues not only disrupt your users but they can also impede proper indexing by Google. And you know, fix them sooner rather than later. You don't want Google thinking your site's getting in a mess or for it to back off its crawling or, even worse, to back off recommending your site is getting in a mess, back off its crawling or, even worse, to back off recommending your site to people in search.

Speaker 1: 8:48

Optimise your server speed If you're starting to see slow response times. This can hurt your SEO. Google likes a fast site. It's also not going to crawl your site as fast as it could do if it's starting to see slow response times. So, again, again, look for those things like caching, image compression, sorting your code out if you've got coding that's causing problems, you know, getting onto a better server, all those kind of things. Watch out for those. It can also be, uh, the kind of thing you need to look at for robotstxt and sitemaps, because if you have an issue with your robotstxt or your sitemap, that could impede Googlebot's ability to crawl and you'll see a number of requests drop or the number of problems rise. So, again, keep an eye on that. And also, yeah, if you're a larger site, it can help you prioritize your crawl budget. If you're seeing that Google's not getting into the pages that you really want, it not getting into the pages that you really wanted to get into, look at you know whether you need to be changing your internal linking or putting no index tags on pages, or changing your robotstxt rules to guide googlebot to the areas that you wanted to go into.

Speaker 1: 9:53

So you know, just to sort of finish up on, this is quite a short one, I think, but you know, just to summarize, you know the google stats report in google search console. It's a really essential tool for any website owner or seo to get a handle on, because it just provides such great insights into how google interacts with your site, revealing that crawl frequency and server performance, any potential issues you might have and regular monitoring of this report and, acting on on the results of the report, you know you can improve your site's crawlability, manage your crawl budget more effectively and you know that's ultimately going to help get Google focusing on the pages you want focusing on and boosting the performance of them. So I hope, if you've not heard of it, that this is something that's you know, brought to your attention. It's another tool in your arsenal, and a free one as well. If you that's you know, brought to your attention. There's a it's another tool in your arsenal and a free one as well. If you um have never used google search console, I highly strongly recommend it. Get yourself validated for your sites on there, because it's just such a wealth of data that if it's being given to you for free by google. Why would you not go there and look at it anyway?

Speaker 1: 10:55

That's it for today, so I hope if you found that useful, you know, please do rate and review us. Go to rate thisispodcastcom SEO. There's links there for reviewing the podcast. It really helps, you know. If you can do it, I'd really appreciate it all. Every review helps us reach more people. And yeah, until next time, just keep optimizing, stay curious and remember SEO is not that hard when you understand the basics. Thanks for listening. It means a lot to me.

Speaker 1: 11:18

This is where I get to remind you where you can connect with me and my SEO tools and services. You can find links to all the links I mentioned here in the show notes. Just remember, with all these places where I use my name, that Ed is spelled with two Ds. You can find me on LinkedIn and Blue Sky. Just search for Ed Dawson on both.

Speaker 1: 11:33

You can record a voice question to get answered on the podcast. The link is in the show notes is in the show notes. You can try our SEO intelligence platform, keywords People Use at keywordspeopleusecom, where we can help you discover the questions and keywords people are asking online. Poster those questions and keywords into related groups so you know what content you need to build topical authority. And finally, connect your Google Search Console account for your sites so we can crawl and understand your actual content, find what keywords you rank for and then help you optimize. Continually refine your content and targeted, personalized advice. Keep your traffic growing. If you're interested in learning more about me personally or looking for dedicated consulting advice, then visit wwweddawsoncom. Bye for now and see you in the next episode of seo is not that hard.