Add Custom Robots.txt File in Blogger

Are you a modern non-technical blogger who wants to increase your blog's ranking and audience but you don't know how to do it and your blogger friends tell you that by editing bots you can get more traffic to your website? Text file? Or don't want search engine crawlers crawling your pages? Or do you have a technical background but don't want to risk changing without subject matter expertise? Either way, this is the place for you. This guide will show you how to add a custom Robots.txt file to Blogger in a few simple steps.

But before we open up the Robots.txt file and get to work, let's take a quick look at what this means:
Warning! Use with caution. Improper use of these features can cause search engines to ignore your blog.

What is Robots.txt?

With every blog you create or post to your website, Blogger automatically creates an associated Robots.txt file. The purpose of this file is for incoming bots (spiders, crawlers, etc. sent by search engines like Google, Yahoo) to learn about your blog, its structure and whether or not to crawl your blog pages. As a blogger, you want some pages on your website to be indexed and crawled by search engines, while other pages don't want to be indexed, e.g. tag pages, demo pages, or other irrelevant pages.

How do you view Robots.txt?

Well, Robots.txt is the first thing this spider sees after entering your website. Your robots.txt file is like an hourly flight attendant, guiding you to your seat and constantly checking that you're not entering any private areas. Therefore, all incoming spiders will only index the files pointed by Robots.txt and prevent others from being indexed.

Where is Robots.txt located?

You can easily view the Robots.txt file in your browser by adding /robots.txt to your blog's address, e.g. B. http://myblog.blogspot.com/robots.txt , or simply go to your blog and go to Settings > Search Engine Preferences > Crawlers & Indexing and select Edit next to Custom Robots.txt.

What does Robots.txt look like?

If you haven't edited your robots.txt file, it will look something like this:

User Agent: Mediapartners-Google
To prevent:
User agent: *
Forbidden: /request
Allows :/
Sitemap: http://myblog.blogspot.com/feeds/posts/default?orderby=UPDATED

Don't worry if it's not colored or there are no line breaks in the code, I've colored it and added line breaks so you can understand what the words mean.

User Agent: Google-Media Partner:
Mediapartners-Google is a Google AdSense crawler which frequently crawls your site to display relevant ads on your blog or website. If you don't allow this option, they won't be able to see ads on the posts or pages you specify. If you are not using Google AdSense ads on your site, delete these two lines.

User agent: *
Those of you with some programming experience will have already guessed the symbolic nature of the '*' (wildcard) character. For others, it dictates that this section (and the lines below) are yours for all incoming spiders, bots, and crawlers.

Forbidden: /request
Do not allow keywords, specify what you are not allowed to do for your blog. Add /search next to it, and that means you're telling crawlers not to crawl your site's search /results pages. So result pages like http://myblog.blogspot.com/search/label/mylabel will never be crawled or indexed.

Allows :/
Let the keywords dictate what to do for your blog. Adding a "/" means robots can crawl your homepage.

Page index:
The keyword -sitemap refers to our blog's sitemap; The code provided here prompts the bot to index every new message. By providing links, we optimize them for efficient inbound crawling, where inbound bots find their way to all of our blog post links and ensure that none of our blog posts are overlooked, from an SEO standpoint.

However, by default, the crawler only indexes 25 posts . So if you want to increase the number of index files, replace the sitemap link with this:

Sitemap: http://myblog.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

And if you have more than 500 published articles, you can use the two maps below:

Sitemap: http://myblog.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: http://myblog.blogspot.com/atom.xml?redirect=false&start-index=500&max-results=1000

How do I prevent my posts/pages from being indexed and crawled?

If you don't already know, here's how to prevent spiders from crawling and indexing specific pages or posts:

Block specific messages

Disallow: /yyyy/mm/post-url.html

The /yyy/mm part indicates the year and month the blog post was published, and /post-url.html is a page that shouldn't be crawled. To prevent articles from being indexed/crawled, simply copy the URL of your article you wish to exclude from indexing and remove the blog address from the start.

Block specific pages

To ban specific pages, you can use the same method as above. Just copy the site url and remove your blog address so it looks like this:

Disallow: /p/page-url.html

Custom Robots.Txt Plugin on Blogger

Now let's see how to add a proper custom Robots.txt file to Blogger :

1. Login to your blogger account and click on your blog.
2. Go to Settings > Search Settings > Web Crawler & Indexing.

3. Select "Edit" next to "Customize robots.txt" and check the "Yes" box.
4. Paste your code or make changes as needed.

5. When finished, click the Save Changes button.
6. And congratulations, you're done!

How can I see if there have been any changes in Robots.txt?

As explained above, simply enter your blog address in your browser's URL bar and add /robots.txt to the end of your URL, as you can see in the example below:

http://helplogger.blogspot.com/robots.txt

After you enter your robots.txt file, you'll see the code to use in your custom robots.txt file. See screenshot below:

Last words:

Are we done with bloggers? Have you finished adding the Custom Robots.txt file to Blogger? It's easy when you know what the keywords mean. If you didn't get it the first time, just follow the tutorial again and you'll be customizing your friend's robots.txt file in no time.

After all, making small changes to your robots.txt file is important when it comes to SEO and website ranking, so don't be lazy. Learning is fun as long as it's free, right?