If you’ve submitted a page for indexing but you are blocking Google bots from crawling it, there’s a pretty obvious issue there – you’re sending mixed messages to Google’s bots and they’re unsure whether you want those pages indexed or not.
These errors should be prioritised because a page that cannot be indexed is of no value to you. Thankfully, correcting this isn’t too hard.
Why is this error triggered?
The robots.txt file, a set of instructions for the Google bot on how to treat your page, tells Google not to index your page. Whilst, most of the time, the Google bot will listen to your instructions and not index the page, you may find that sometimes the page will be indexed as the bot will find it relevant.
Whether your page is showing up in the search results or not, it’s important to fix this error to prevent any issues in the future.
How to stop URLs from being blocked by robots.txt?
First off, you should use the robots.txt tester to double check that there’s an issue – it’s as simple as selecting your search console account and your robots.txt will be displayed along with the number of errors and warnings. You can also enter in a specific URL to test if it’s blocked.
After you’ve identified which URLs are being blocked (you can pull these from Search Console), you’re going to need to change the robots.txt file to allow indexing. Don’t worry, it’s not as hard as it sounds – just follow these steps:
- Open up the robots.txt tester and enter your URL, then press search
- If the robots.txt file is blocking Google from indexing your page, it should look like this:
User-agent: Googlebot
Disallow: /nogooglebot/
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
- Remove Googlebot as the user-agent and remove /nogooglebot/
- Submit the new robots.txt file to Google – job done
Note – On the other hand, if you want to ensure that your page isn’t indexed at all, you shouldn’t use a robots.txt to instruct Google as it won’t always follow your instructions. Instead, you should use a ‘noindex tag’. Do note that pages with a noindex attribute will eventually lead to links on that page being labelled as nofollow, meaning that link equity will not be passed onto those pages.
Need help with your robots.txt file?
You can request a free technical SEO quote by filling out the form on the bottom of the technical audit service page if all of this seems too technical for you or if you’d like to have a comprehensive technical audit of your site performed.