Cloaking - a Blackhat story

Hello and welcome to. Seo is not that hard. I'm your host, ed Dawson, the founder of keywordspeopleusecom, the place to find and organise the questions people ask online. I'm an SEO developer, affiliate marketer and entrepreneur. I've been building and monetising websites for over 20 years and I've bought and sold a few along the way. I'm here to share with you the SEO knowledge, hints and tips I've built up over the years. With you, the SEO knowledge, hints and tips I've built up over the years.

Hello, welcome to episode 96 of SEO is not that hard, and today I'm going to be talking about cloaking. This is not something you might hear talked about by a lot of mainstream SEOs because essentially, it's a black cat technique. Now, I am not telling you this um in any way suggesting that you do it. Um, it's interesting in terms of learning how certain black cat techniques work, because I think any rounded seo needs to know what to look out for if they come across this kind of behavior. Or if you are dealing with someone who is suggesting that you do something and you need to know whether it's Black Cat or not, and anyone who wants to do Black Cat stuff knowing full well what the risks and the rewards are, then that's fine with me. I don't judge. It's not something I do anymore. I've just talked about plenty of times Because for me, I still believe that now I want to build for the long term, that now I want to build for the long term, I'm building for the long term means not doing anything that's going to make overtly risky in terms of getting all my hard work destroyed by a Google update or manual penalty or something like that. So, with that disclaimer given, let's talk about cloaking.

So what is cloaking in its simplest forms? Well, that is that you, essentially, you show Google one version of your website or web page and you show other people a different version. Why would you want to do this? For example, say, you want to rank something and you want to google to think it's really nice and clean and nice, clean content, but it might not necessarily be content that's going to monetize very well or get people to take an action that you want them to. That you might think Google might not like what the content of the actual page you want to show people is. So I'll show them some really clean page with really clean content and hopefully rank based on that content, because that's what Google and Googlebot will see. Then, when people click to me from the search engine result pages, they'll get a different version which gets them to do something completely different. Say, google obviously doesn't like this because it wants to show people in the search engine results based on what they're actually going to see when they land there. That's essentially what cloaking is, so it's showing a different version of the site to Google than you show to everybody else.

The story, a black hat story. If we go back to the years of 2008, 2009, this was before. I was very clean in what I do seo and one of the things I was doing a lot of the time was buying expired domains to put content on to then link to mine and other people's websites. And you know it was. It was. It was not as black as it is now, say, it was more gray, but anyway it was.

We knew that it was against terms of service, but at the time google just couldn't detect these kind of links. It wasn't sophisticated enough to detect that kind of link building, but it was one of those ones where, if your competitors reported you and reported those links, google would hand out manual penalties to people. So the key to doing that kind of work was you wanted to avoid a manual penalty so that you could just get on for the rest of your life and build the links like that. So we wanted to avoid being detected. How do how people detecting clinks and reporting them back then? Well, they were using clink reporting tools such as, I mean, the main ones were at the time, yahoo. You could get quite good backlink information out of Yahoo and Moz. They had a backlink explorer and I think there was Majestic around back then. There was a few, but Yahoo and Moz were really the big ones.

So what we wanted to do was we wanted to keep our links and links we're building from this method out of those tools, but available to Google. So what we did was we cloaked our entire website. But rather than cloaking them against Google, we cloaked them against Yahoo and against and against mars. I think it was might have been scm rush the scm rush bot. There's quite a few. Any other, basically every bot other than google bot. We cloaked the site.

So we did it in a few ways. We we gave um different versions of the robotstxt. So googlebot's a very clean robotstxt that's sort of allowed to go wherever it wanted, whereas we'd show everyone else a much more restrictive robotstxt. But we also didn't trust that some crawlers and bots would respect our robotstxt, so we then would also cloak the entire site against any kind of bot. We also cloaked from the search engine results pages because obviously you could at the time, like you can now you can, google will give you some links. If you type link in a page in the search engine results, it would report some of the links you knew about. So we also cloaked anyone that came from a google search results page to look at one of our pages. We would remove the links. We keep the content there, but we would remove the links. So we essentially used cloaking not to show google something different to what we were showing most people, but it was just we're trying to show it cloak. Cloak it differently for those sites that could have found those links that would enable someone to have reported these networks of sites. This was basically pbns at the time, public blog networks. We were creating um and we use clooki in that way.

On a technical level, the way Floki works is it relies on interrogating the user agent for any request that's made to your web server. So you would use some software that would intercept every request that came to the web server and it would look at the user agent and if that user agent was Googlebot, then we'd do one thing user agent. And if that user agent was googlebot, then we do one thing. If it was different kinds of other um user agents, we might make other decisions. And again, we'd also look at the referring url. So if it was coming from google, then we knew that someone had clicked in google to find that to come to this page and we would then take actions depending on that. So you can you can cloak, say, on any information that really comes in that in that header, so that could be a referrer and it could be the user agent and would also look at the ip address, because obviously sometimes google will check a page, check a site, without using google as the header um, at least that's what we believed. You know I would have to have not. It's been so long since I've done any of this stuff that I don't know whether google still do that or whether they did do that, whether it was just our fear that they did that.

But we actually also built a database of um, genuine google ips, um. So these are ips address addresses that we knew googlebot used and we um also shared that with other people who were doing similar things. So we built up a um database of ip. So if we ever saw googlebot come from an ip and we used to do reverse proxy sorry, reverse ip lookups to see where they came from, because google at the time we're probably still is was only um um coming from ip addresses that are registered in Mountain View in California. So you could then go back and say, right, if any IP address that's come from Mountain View, we would treat it as if it was Googlebot or Google and we'd serve it that way. Our aim was also, if we've got any kind of manual review, that we would show a manual user from Google. This is what they were getting. So that's how you actually actually do it.

So how do you detect, if you can, if you're suspicious that a site might be cloaking, how can you um, you know, as a genuine, normal person check this? Well, again, it's going to depend on how sophisticated they've done it. I've just given you a whole details of how we did it, and we went to quite a sophisticated level. A lot of places go slightly less sophisticated. They might just look for Googlebot being the user agent and not do any of the IP checks. And you can detect those by if you can get a plugin for your browser which allows you to change the user agent that Chrome sends when it makes requests to the web server. And they've got all sorts of choices like Bingbot, googlebot you can just do some manual ones or you can enter your own user string and change the user agent. So a lot of the time, if you find one of these, if you just change your user agent in Chrome to be Googlebot, you might find that a site would show you a different version and that's how you can spot how they do it. But again, it all just depends on how far down the line of how sophisticated they've made their cloaking, how far they're going to try to detect whether it's true Googlebot or someone faking Googlebot, before they'll actually show you the cloaked version. So, yeah, so I hope that's been a sort of useful explanation.

And, as I say this, all this stuff is completely against google's terms of service. It's against all the guidelines. If you do this and get caught, you will get banned to high heaven, um. So I wouldn't suggest anyone does it, but I think it's important for people to understand how it does work, so they can, so that you can guard against it, so that you can see if anyone ever comes to you and says we've got this fantastic technique and this is what we'll do, you can just say, no, I don't want to do that at all and run for the high offense. Or if your risk reward says, hey, I want to try all the things, then you know at least you will know the risks around doing it. And obviously, just to reiterate, from my side, this is nothing that I've done since, uh, penguin, basically since penguin killed all this for me, and it's not anything I've done for for many, many years. So you know, it's not a technique I'm using now and it's definitely not a technique that I recommend. So, yeah, hope you find it useful.

Before I go, I just wanted to let you know that if you'd like a personal demo of our tools that keywords people use, that you can book a free, obligation one-on-one video call with me where I show you how we can help you level up your content by finding and answering the questions your audience actually have. You can also ask me any SEO questions you have. You just need to go to keywordspeopleusecom slash demo where you can pick a time and date that suits you for us to catch up Once again. That's keywordspeopleusecom slash demo and you can also find that link in the show notes of today's episode. Hope to chat with you soon. Thanks for being a listener. I really appreciate it. Please subscribe and share. It really helps.

Seo is not that hard. It's brought to you by keywordspeopleusecom, the place to find and organize the questions people ask online. See why thousands of people use us every day. Try it today for free at keywordspeopleusecom To get an instant hit of more SEO tips. Then find the link to download a free copy of my 101 quick SEO tips in the show notes of today's episode. If you want to get in touch, have any questions, I'd love to hear from you. I'm at channel5 on Twitter. You can email me at podcast at keywords people usecom. Bye for now and see you in the next episode of SEO is not that hard.