How can you setup Custom Robot.txt in BlogSpot blogs for better SEO?

You might have heard about the Robot.txt in Google Search console (Previously Webmaster’s Tools). In this post, I am going to tell you how to use “Robot.txt” file in blogger.com blogs for getting best from search engines.  Robot.txt is mainly for the benefit for the bloggers so it is known as Custom Robot.txt file.

The use of this file needs complete information by its users. Users must have complete information about the keywords used in a file. I am here to make you familiar and provide you with complete information about Robot.txt and different keywords and tags that are used.

What is a Robot.txt file?

Robot.txt file is not very complicated to be understood. It is very simple. It is basically a text file that contains few lines of simple codes. Every person saves this file on the root of their website or blog’s server. Robot.txt file restricts certain posts. The search engines always scan the robot.txt file and then show the instruction according to it. It is very helpful in many ways like you do not want the search engine spiders to index your particular page or any page that is not useful to you then it won’t show you the unnecessary things it will always point out the things that are needed by you.

Understanding different keywords and tags of robot.txt

When you will open the robot.txt you will find various keywords and tags so here I am to explain some of them.

User-agent: Media partners –Google

Disallow:

User-agent:*

Disallow:/search

Allow: /

Sitemap:

http://example.blogSpot.com/feeds/postsorderby=UPDATED

If a person wants to work on the blog platform he needs to understand the meaning of various keywords with their functioning. So now I am going to explain each of them in a bit of detail.

Explanation of line 1 and 2

User-agent: Media partners-Google

Disallow:

These codes are basically used and helpful for those bloggers who are using Google Adsense Ads on their blogs. These codes help the Google Adsense Ads to cast their ads in proper and a better way. If you are not able to use Google Adsense then it is better to leave the platform of the blog.

Explanation of line 3:

“ User – agent:*”

User – agent:*

If we see in the terms of programming then an asterisk (*) this sign basically means allowing all. But when we use this sign for the blog purpose then this means that we are inviting all kind of robots to visit our blog.

These keywords are very important for blogs because they act as a guard. They allow the search engine to search specific or general information.

Explanation of line 4:

“disallow:/search”

So these codes are directly added by the bloggers to the robot.txt folder which restricts the search engine robot from viewing all links which are related to keyword “search” after the domain name. For example http://www.geteverything.org/search/label/SEO-tips

Taking this example in this example the search engine will not be able to show any result when a person will click on the above link because it includes the keyword “search” after the domain name.

Use “disallow:” keyword for blocking Search engine Bot’s from crawling some specific posts?

Starting from keyword “disallow” it is a very important keyword. It is used to hide various posts and pages from search engines.

Hiding a particular post from search engines

If you do not want to share or show some post then they can be hidden by following the format

Disallow: /yyyy/mm/your-post-url. In this code, you can replace “yyyy” with the year and “mm” with the month and you are able to hide the particular post you want to.

For example, if you want to hide a post which is located in http://www.geteverything.org/2013/06/blogger-custom-robort-txt-setup.html then you can hide the post by pressing Disallow:/2013/06/blogger-custom-robot-txt-setup.htm. but make sure you do not add the domain name in the Raobot.txt file.

Hiding a particular page from search engines

In above method, we learned about hiding a post. Now we will see how to hide a particular page

You can use format Disallow:/p/your-page-URL.  In the place of “your-page-url, you can type the particular page you want to hide.

For example

If you want to hide a page located at http;//www.geteverything.org/p/our-services.html, you can use the above format and add a robot.txt file to hide it.

What will happen after removing “disallow:/search” in the robot.txt file?

If keyword “disallow” will be removed then the search engine robot will crawl and index everything.

Understanding the 5th line: “Allow: /”

This code is the main keyword in the blog platform. In case the “disallow” code has been removed by you then you can search in the engine will crawl all kind of link on your blog.

Understanding the last line: sitemap

http://example.blogSpot.com/feeds,postsorderby=UPDATED

This format tells that the search engine crawls about only 25 posts.

How to instruct the search engine bots to crawl 500 recent posts on your blog?

Use can use the formats like http://yourblog.blogspot.com/feeds/postsorderby=UPDATED

Or use can use https://yourblog.blogspot.com/atom.xml?redirect=false&start to instruct the search engine.

How can you instruct the search engine bots to crawl 2000 recent posts on your blog?

You can use the codes to instruct the search engine like

http://yourblog.blogspot.com/feeds/postsorderby=UPDATED

OR

http://yourblog.blogspot.com/atom.xml?redirect=false&star-

in the above codes replace “yourblog.blogspot.com with URL of your blog

How to setup custom Robot.txt file in blogger.com blogs?

• First, you have to sign in to your blogger account.
• In dashboard then you are to click on settings.
• When you open the settings you are to click on “search preference option. And then a new screen will appear you have to click on edit link which is beside “disabled” option of “custom robot.txt.
• Now you have to enable the option by pressing on yes button.

Now as you have learned about tags and keywords you are to apply them.

How to check the contents of your Robot.txt file without logging into blogger?

Robot.txt file is the root of blogger. So this is the reason why you can easily check the content by visiting

http://www.yourblog.blogspot.com/robot.txt

or http://www.yourblog.com/robot.txt

Editor’s desk:

Now by reading above we can see how keywords and tags are important at the blog platform. The explanation above is very easy to understand and helpful too.

---You may also like---

List of 50 CommentLuv Enabled Blogs 3-5 PR in Technology Niche 2015-16 How to get .edu Backlinks & List of 25 Comment Enabled edu Blogs Importance of Search Engine Optimization (SEO) & Its Cost 4 Rarely Used Ways to Get Free Whitehat Backlinks Google PageRank Update May 2012