Perfect robots.txt file for blogger:  Robots.txt file is very important for the SEO of your blog. In this post, I have shared how you can create perfect robots.txt file for blogger. 

create perfect robots.txt file for blogger 

create perfect robots.txt file for blogger

What is robots.txt file for blogger


robots.txt is a file that is in the root folder of your website.

A robots.txt file is a set of instructions for bots

The robots.txt tells the google search engine bots to, which parts of the website to crawl and index and which parts not

If you make any mistake while setting robots.txt file, then Google Bots will not crawl your site and your site will not appear in the search results.

Don’t make any mistake while creating a robots.txt file

Why Robots.txt file for blogger is important?


The Robots.txt file for blogger is what tells the search engines which pages to access and index on your website or which pages not to.

For example, if you set in your robots.txt file that you don't want the search engines to be able to access your About page, that page won't be able to show up in the search results and web users won’t be able to search about the page in your website or in any search engine.

Create perfect robots.txt file for blogger 

create perfect robots.txt file for blogger

The basic robots.txt file looks like this 
User-agent : *
Disallow: /

The '*' mark after User-agent means that robots.txt file allows/tells the web bots to visit the site.

The  '/' slash after the Disallow tells the robot to not visit any pages on the site. 

User-agents : 

Each search engine identifies itself with its user agent. You can set custom instructions for each search engine in your robots.txt file. there are 100+ user-agents, but here are some useful ones for SEO of your blog.

  • Google: Googlebot 
  • Google-images: Googlebot-Image
  • Bing: Bingbot
  • DuckDuckgo: Duckduckbot
  • Yahoo: Slurp
  • Baidu: Baiduspider 

Robots.txt file for blogger: type-1

       

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Disallow: /b
Allow: /

Sitemap: https://www.techstuff.in/sitemap.xml
       
 
Note: Don't forget to change the techstuff.in with your blog domain or address

This is a simple basic robots.txt file for blogger. 

Robots.txt file for blogger: type-2

       

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Disallow: /b
Allow: /

Sitemap: https://www.techstuff.in/feeds/posts/default?orderby=updated
       
 
Note: Don't forget to change the techstuff.in with your blog domain or address

If you want search engine bots to crawl the most recent 500 posts, then you should need to use this robots.txt file 

Robots.txt file for blogger: type-3

       

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Disallow: /b
Allow: /

Sitemap: https://www.techstuff.in/atom.xml?redirect=false&start-index=1&max-results=500
       
 
Note: Don't forget to change the techstuff.in with your blog domain or address

If you already have about 500 posts on your website, then  I recommend you to use type-3 of robots.txt file.

For more than 500 posts use the following robots.txt file. 

 
User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Disallow: /b
Allow: /

Sitemap: https://www.techstuff.in/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: https://www.techstuff.in/atom.xml?redirect=false&start-index=501&max-results=500


About the post : 
I shared create perfect robots.txt file for blogger - SEO robots.txt article to help bloggers to create robots.txt file 

Our Website : www.techstuff.in

If your friends need help with creating perfect robots.txt file for blogger, please share this post with them. 
If you are a blogger, Mention this post in your blog post. 

Tag : create perfect robots.txt file for blogger.

Also read: BEST SEO TIPS

read more:  Are you looking to monetize your website traffic? read below 

4 Comments

  1. thank you so much techstuff.in for helping me to create a robots.txt file

    ReplyDelete
  2. SHUKRIYA BHAAIJAAN AAP JO BHI HO HELP HO GAYI MERI��

    ReplyDelete
  3. Very nice information 👍 great work

    ReplyDelete

Post a Comment

Previous Post Next Post