Add Custom Robots.Txt File In Blogger

Are you one of modern day bloggers without much knowledge of technical details, looking to enhance your blog's ratings and audience but don't know how to and your blogger friend told you that you can get a hike into your site visitors via editing your Robots.txt file? Or maybe you don't want search engines spiders to crawl through your pages? Or you do have a technical background but don't want to risk making changes without expert's words on topic? Well in either case, this is the right place for you to be. In this tutorial, you will see how to add Custom Robots.txt file in Blogger in a few easy steps.

will index only 25 posts, so if you want to increase the number of index files, then replace the sitemap link with this one:

Sitemap: http://myblog.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

And if you have more than 500 published posts, then you can use these two sitemaps like below:

Sitemap: http://myblog.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: http://myblog.blogspot.com/atom.xml?redirect=false&start-index=500&max-results=1000

How to prevent posts/pages from being indexed and crawled?

In case you haven't yet discovered yourself, here is how to stop spiders from crawling and indexing particular pages or posts:

Disallow Particular Post

Disallow: /yyyy/mm/post-url.html

The /yyy/mm part specifies your blog posts publishing year and month and /post-url.html is the page you want them not to crawl. To prevent a post from being indexed/crawled simply copy the URL of your post that you want to exclude from indexing and remove the blog address from the beginning.

Disallow Particular Page

To disallow a particular page, you can use the same method as above. Just copy the page URL and remove your blog address from it, so that it will look something like this:

Disallow: /p/page-url.html

Adding Custom Robots.Txt to Blogger

Now let's see how exactly you can add Custom Robots.txt file in Blogger:

1. Sign in to you blogger account and click on your blog.
2. Go to Settings > Search Preferences  > Crawlers and indexing.

Advertiser

3. Select 'Edit' next to Custom robots.txt and check the 'Yes' check box.
4. Paste your code or make changes as per your needs.

5. Once you are done, press Save Changes button.
6. And congratulations, you are done!

How to see if changes are being made to Robots.txt?

As explained above, simply type your blog address in the url kafetaria of your browser and add /robots.txt at the end of your url as you can see in this example below:

http://helplogger.blogspot.com/robots.txt

Once you visit the robots.txt file, you will see the code which you are using in your custom robots.txt file. See the below screenshot:

Final Words:

Are we through then bloggers? Are you done adding the Custom Robots.txt in Blogger? It was easy, once you knew what those code words meant. If you couldn't get it for the first time, just go again through the tutorial and before long, you will be customizing your friends' robots.txt files.

In any case, from SEO and site ratings it's important to make that tiny bit of changes to your robots.txt file, so don't be a sloth. Learning is fun, as long as its free, isn't it?