You might have heard about the Robot.txt in Google Search console (Previously Webmaster’s Tools). In this post, I am going to tell you how to use “Robot.txt” file in blogger.com blogs for getting best from search engines. Robot.txt is mainly for the benefit for the bloggers so it is known as Custom Robot.txt file.
The use of this file needs complete information by its users. Users must have complete information about the keywords used in a file. I am here to make you familiar and provide you with complete information about Robot.txt and different keywords and tags that are used.
- What is a Robot.txt file?
- Understanding different keywords and tags of robot.txt
- Explanation of line 1 and 2
- Explanation of line 3:
- Explanation of line 4:
- Use “disallow:” keyword for blocking Search engine Bot’s from crawling some specific posts?
- Hiding a particular post from search engines
- Hiding a particular page from search engines
- What will happen after removing “disallow:/search” in the robot.txt file?
- Understanding the 5th line: “Allow: /”
- Understanding the last line: sitemap
- How to instruct the search engine bots to crawl 500 recent posts on your blog?
- How can you instruct the search engine bots to crawl 2000 recent posts on your blog?
- How to setup custom Robot.txt file in blogger.com blogs?
- How to check the contents of your Robot.txt file without logging into blogger?
What is a Robot.txt file?
Robot.txt file is not very complicated to be understood. It is very simple. It is basically a text file that contains few lines of simple codes. Every person saves this file on the root of their website or blog’s server. Robot.txt file restricts certain posts. The search engines always scan the robot.txt file and then show the instruction according to it. It is very helpful in many ways like you do not want the search engine spiders to index your particular page or any page that is not useful to you then it won’t show you the unnecessary things it will always point out the things that are needed by you.
When you will open the robot.txt you will find various keywords and tags so here I am to explain some of them.
User-agent: Media partners –Google
If a person wants to work on the blog platform he needs to understand the meaning of various keywords with their functioning. So now I am going to explain each of them in a bit of detail.
Explanation of line 1 and 2
User-agent: Media partners-Google
These codes are basically used and helpful for those bloggers who are using Google Adsense Ads on their blogs. These codes help the Google Adsense Ads to cast their ads in proper and a better way. If you are not able to use Google Adsense then it is better to leave the platform of the blog.
Explanation of line 3:
“ User – agent:*”
User – agent:*
If we see in the terms of programming then an asterisk (*) this sign basically means allowing all. But when we use this sign for the blog purpose then this means that we are inviting all kind of robots to visit our blog.
These keywords are very important for blogs because they act as a guard. They allow the search engine to search specific or general information.
Explanation of line 4:
So these codes are directly added by the bloggers to the robot.txt folder which restricts the search engine robot from viewing all links which are related to keyword “search” after the domain name. For example http://www.geteverything.org/search/label/SEO-tips
Taking this example in this example the search engine will not be able to show any result when a person will click on the above link because it includes the keyword “search” after the domain name.
Use “disallow:” keyword for blocking Search engine Bot’s from crawling some specific posts?
Starting from keyword “disallow” it is a very important keyword. It is used to hide various posts and pages from search engines.
Hiding a particular post from search engines
If you do not want to share or show some post then they can be hidden by following the format
Disallow: /yyyy/mm/your-post-url. In this code, you can replace “yyyy” with the year and “mm” with the month and you are able to hide the particular post you want to.
For example, if you want to hide a post which is located in http://www.geteverything.org/2013/06/blogger-custom-robort-txt-setup.html then you can hide the post by pressing Disallow:/2013/06/blogger-custom-robot-txt-setup.htm. but make sure you do not add the domain name in the Raobot.txt file.
Hiding a particular page from search engines
In above method, we learned about hiding a post. Now we will see how to hide a particular page
You can use format Disallow:/p/your-page-URL. In the place of “your-page-url, you can type the particular page you want to hide.
If you want to hide a page located at http;//www.geteverything.org/p/our-services.html, you can use the above format and add a robot.txt file to hide it.
What will happen after removing “disallow:/search” in the robot.txt file?
If keyword “disallow” will be removed then the search engine robot will crawl and index everything.
Understanding the 5th line: “Allow: /”
This code is the main keyword in the blog platform. In case the “disallow” code has been removed by you then you can search in the engine will crawl all kind of link on your blog.
Understanding the last line: sitemap
This format tells that the search engine crawls about only 25 posts.
How to instruct the search engine bots to crawl 500 recent posts on your blog?
Use can use the formats like http://yourblog.blogspot.com/feeds/postsorderby=UPDATED
Or use can use https://yourblog.blogspot.com/atom.xml?redirect=false&start to instruct the search engine.
How can you instruct the search engine bots to crawl 2000 recent posts on your blog?
You can use the codes to instruct the search engine like
in the above codes replace “yourblog.blogspot.com with URL of your blog
How to setup custom Robot.txt file in blogger.com blogs?
- First, you have to sign in to your blogger account.
- In dashboard then you are to click on settings.
- When you open the settings you are to click on “search preference option. And then a new screen will appear you have to click on edit link which is beside “disabled” option of “custom robot.txt.
- Now you have to enable the option by pressing on yes button.
Now as you have learned about tags and keywords you are to apply them.
How to check the contents of your Robot.txt file without logging into blogger?
Robot.txt file is the root of blogger. So this is the reason why you can easily check the content by visiting
Now by reading above we can see how keywords and tags are important at the blog platform. The explanation above is very easy to understand and helpful too.