What is the Important of Robots.txt ?

the success of any professional blog usually depends on how the Google search engine ranks your blog. We store a number of posts/pages/directories in our website structure. Often we don't want Google to index all of these components. For example, you might have a file for internal use and it's useless to search engines. You don't want this file to appear in search results. Therefore, it is wise to hide such files from search engines.


The robots.txt file contains directives that are respected by all the top search engines. Using this directive, you can give instructions to the web spider to ignore certain parts of your website/blog.

Custom Robots.txt for Blogger/Blogspot/website

Blogger/Blogspot is a free blogging service, the robots.txt of your blog is not directly under your control. But now Blogger has made it possible to make changes and create Custom Robots.txt for each blog. Robots.txt for Blogger/Blogspot blogs usually looks like this:

Add Custom Robots.txt File in Blogger/Blogspot

Go to your blogger dashboard

Go to Settings > Search Preferences > Crawlers and indexing > Custom Robots.txt > Edit > Yes

This is how you can make changes to the robots.txt file

User-agent: Mediapartners-Google

Disallow:

User-agent: *

Disallow: /search

Allow: /

Sitemap:https://buisblogging.blogspot.com/sitemap.xml

After making changes, click the Save Changes button.