Google Search Console: Submitted URL blocked by robots.txt

What issue is this page dealing with?
If you are faced with the issue of a page being blocked by robots.txt, it needs to be dealt with promptly so google can get back to indexing your pages. In this page we will explore why this error is triggered and how to stop urls being blocked by using the robots.txt tester.

Want a free audit?

Send me an email with your URL and I’ll spin up a top level SEO audit, free of charge. If you want a slightly more advanced one; just add me to your Google Search Console (email is the same). 

What I cover here

If you’ve submitted a page for indexing but you are blocking Google bots from crawling it, there’s a pretty obvious issue there – you’re sending mixed messages to Google’s bots and they’re unsure whether you want those pages indexed or not.

These errors should be prioritised because a page that cannot be indexed is of no value to you. Thankfully, correcting this isn’t too hard.

Why is this error triggered?

The robots.txt file, a set of instructions for the Google bot on how to treat your page, tells Google not to index your page. Whilst, most of the time, the Google bot will listen to your instructions and not index the page, you may find that sometimes the page will be indexed as the bot will find it relevant.

Whether your page is showing up in the search results or not, it’s important to fix this error to prevent any issues in the future.

How to stop URLs from being blocked by robots.txt?

First off, you should use the robots.txt tester to double check that there’s an issue – it’s as simple as selecting your search console account and your robots.txt will be displayed along with the number of errors and warnings. You can also enter in a specific URL to test if it’s blocked.

After you’ve identified which URLs are being blocked (you can pull these from Search Console), you’re going to need to change the robots.txt file to allow indexing. Don’t worry, it’s not as hard as it sounds – just follow these steps:

  1. Open up the robots.txt tester and enter your URL, then press search
  2. If the robots.txt file is blocking Google from indexing your page, it should look like this:

      User-agent: Googlebot

      Disallow: /nogooglebot/

      User-agent: *

      Allow: /

      Sitemap: https://www.example.com/sitemap.xml

  1. Remove Googlebot as the user-agent and remove /nogooglebot/ 
  2. Submit the new robots.txt file to Google – job done

Note – On the other hand, if you want to ensure that your page isn’t indexed at all, you shouldn’t use a robots.txt to instruct Google as it won’t always follow your instructions. Instead, you should use a ‘noindex tag’. Do note that pages with a noindex attribute will eventually lead to links on that page being labelled as nofollow, meaning that link equity will not be passed onto those pages. 

Need help with your robots.txt file?

You can request a free technical SEO quote by filling out the form on the bottom of the technical audit service page if all of this seems too technical for you or if you’d like to have a comprehensive technical audit of your site performed.

Need CMS specific SEO support?  I’ve got extensive experience working with the below CMS – view each service page to learn more. I’ve also include some additional links which may be useful for that particular CMS.

Would you like a free SEO audit? 

Enter your details in the form here and I’ll run a top level audit of your website, it shouldn’t take long and will give you some basic information on your site’s health. 

In addition to getting a free audit, if you are stuck on a particular problem, do feel free to reach out to me – I’m always happy to help fellow SEO experts answer any questions.