Hey guys, We want to talk to you about robot.txt in particular what it is and how it’s going to apply to you as a publisher with advertisers on your site. So robots.txt. What is it? So it’s a plain text file. You’re going to upload to your site. It’s going to tell crawler such as Google which pages of your site they can index. So really the way that Google works is this going to go to your site. It’s going to read the links on a page.
How index pages
It’s going to follow each of those links. It’s going to be each of those pages then follow those links which is kind of where you hear the term robot says Travis spiders because they’re really crawling the web very witty, we know so what can you use robot.txt to do so when Google first comes your page they’re going to check. Do you have a robot.txt file you’re going to download that that’s going to give them a set of instructions. It’s basically the tell them which Pages they can and cannot index and Google’s a good players. They’re going to listen to those in Structions, they’re going to not index Pages. You tell them not to some examples of this is maybe you tell them don’t index my admin page or other Pages were sensitive information. If you don’t want to appear in Google search results. Okay, great. That sounds awesome. Right. So what’s the problem of robots.txt? Well, the thing is a lot of people think if I can tell robots to not go to certain parts of my sight. Good night. Do this is stop bot traffic. We commonly see that here at media Vine. So what do people often put on the robots.txt file to disallow?
Indexing by Everyone by googlebot thinking. Okay. So Google can come from my page. But all the bad guys can here’s the problem as I mentioned before Google the first thing I do when they get to their site your site is they download that robots.txt, and they’re actually making a conscious choice to not index those pages because they’re following your instructions. That’s because they’re a good actor or a good player in this scenario the bad guys or bot traffic. They don’t have to download that robots up here to steal and in fact, they won’t They’re just going to come to your site. They’re going to do what they were planning on doing. So here’s the problem if you end up putting up a robots.txt file and you’re blocking crawlers from browsing your site. You’re actually going to be blocking the Crawlers of a bunch of advertisers. So on advertisers in particular contextual advertisers are going to want to browse your site to figure out the keywords that you’re writing about. So, for example, if you’re running about sugar a diabetes medicine is not going to want to advertise on your site conversely if you’re running
Using flour maybe a flower company like King Arthur flour does want to advertise your site. So they want those keywords. So good examples would be Google AdSense. So Google has in section one of the biggest buyers here through the Google an exchange program. So Google AdSense is one of the most infamous contextual advertisers. If you are blocking all crawlers besides googlebot, you’re actually gonna be blocking the Google AdSense crawlers or Google AdSense runs a different crawler than Google search engine, which is Google bot. She might in it. Certainly, the blocking all of Google assets from browsing your page. There’s a lot of other smaller tier players and I’m media dotnet but all do contextual and there’s so many more. Well, you’ve seen some of our Publishers in the back Sally blocking these crawlers hurting their cpms by 25 to 50 percent. So this year we’ve really seriously impacting your bottom line. That’s how important are these advertisers are. So we implore you to do here at media Vine is if you don’t know what you’re doing with your robot.txt file.
Just Kickin blank. We’re gonna provide a template at the bottom of the blog post associated with this video that gives you a pretty good easy to use robots.txt file. You should just have your Tech person help you with this and bottom line. If you don’t know what you’re doing in that file,
you’re going to cause a lot more harm than good. So just let the smart Crawlers for you out there doing don’t try to use robots TF dot txt for what it was intended to do.