Understanding & Fixing Indexation Issues

While indexation can be an intimidating topic for many business owners, Google Search Console (also known as GSC) provides great insight into what pages are performing well and can help you zero in on pages that Google is opting to ignore. This insight can help website owners understand what content and features play well with Google and what pages or features may need a bit more work.

GSC is a free tool offered by Google that helps in the maintenance, monitoring, and troubleshooting related to the performance of a site in Google Search results. While it is a free tool, it can provide a wealth of insight that will make you aware of any issues that Google encounters as it crawls and indexes your website. As the most utilized search engine in the world, what more could you want than a free tool where Google tells you what they think of your website?

Bots, Crawling, Indexation, & the Coverage Report

The index coverage report is one of the most insightful tools within Google Search Console.  This report provides you with a listing of errors encountered as well as information on what pages have been excluded from indexation. The term “error” can be a bit alarming, but these errors are communicating any issues that Google had while crawling your site.

Wondering exactly what a crawl is and why Google is conducting this activity on your site? Google conducts these crawls with their bot or crawler, also referred to as a Googlebot. When the bot is crawling you’re your site it’s looking at the information on your web pages and determining if the information is worthy of indexation. When pages are indexed it means they have been stored on index servers, which makes them eligible to come up in search engine results pages.

The great thing about the search engine bots crawling your site is that not only are your pages able to get indexed but through the GSC tool Google gives us some insight into what we can do to have a better rate of indexation. And more indexation means more organic exposure, which we all want!

Harnessing the Power of the GSC Coverage Report

If you are like a lot of website owners, you may be new to utilizing GSC to resolve indexation errors and increase your overall indexation. Finding it can be a bit tricky when you first start, but once you know where it is, you’ll find yourself spending quite a bit of time here.

The directions below are assuming you already have an account, have submitted your sitemap to Google in the past, and you’re logged into Google Search Console.

First, select the website you would like to view the indexation on.  If you have more than one website, it will appear in a drop-down list. Next, click coverage.

You’ve made it to the coverage report!

The first thing you’ll see are the errors. You’ll be able to click on each line item and see what pages are impacted. The report also lists pages that have been excluded from indexation but are not necessarily a result of an error.

If you’re like most first-timers seeing this report, your overarching question may be: NOW WHAT?

Common Google Search Console Errors: What They Mean and How to Resolve Them

We will explore some of the more common issues and how you can resolve them. This is by no means an exhaustive list but is, in TopSpot’s experience, encountered most often:

Not Found (404)

You will see this error when Google attempts to crawl a link and finds that it does not exist. This can happen when a page is removed from a site or redirects are not in place. The best practice is to fix broken links. And if you have removed a page, create a 301 redirect to related content to provide the best user experience.

Soft 404

Google will often provide this message if they feel that the page contained useful content at one time but no longer offers the same experience. When you encounter this message, you may be able to resolve it by creating a redirect to a new page or upgrading the level of content on the page.

Blocked by “noindex” Tag

This happens when Google attempts to index a page and it encounters a “noindex” directive.  If this is a page you didn’t want Google to index, you have been successful! To make your site crawls more efficient, you may want to remove any internal links to this page, as this would keep Google from wasting any time looking at the page.

If you do want the page indexed, you need to remove the “noindex” tag from the page that exists in your source code. 

Blocked by Page Removal Tool

When you encounter this type of exclusion, it likely means that you or someone else at your company asked Google to remove the page utilizing their page removal tool.  This is a temporary solution, so if you want to ensure the page stays blocked, you should consider removing the page from your site or Google can choose to index it again in the future.

Blocked by robots.txt

If you receive a notice that your page cannot be indexed because it is blocked by robots.txt, it means that at some point the page has been added to your robots.txt file as one not to crawl or index.  If you want the page to be crawled, you’ll need to remove it from your robots.txt file, which is located in the backend of your website.

Crawled – Current Not Indexed

If you see this, it means that the specific pages were crawled by Google, but Google chose not to index them, and they may or may not be indexed in the future.

When you see this, it’s important to take a good look at the content that lives on the page.  Does the information provide any valuable insight?  Is the page easy to interact with?  Do you link to helpful information within your site or external sites?  Is the page optimized in a way that gives Google a clear indication of what type of information they should expect to find on the page?  Do your page titles, H1’s, and content align?

Often improving the content on the page, and ensuring it is optimized, will increase the chance of Google choosing to index the page in the future.

Alternate Page with Proper Canonical Tag

When you see this message, it means that the page is a duplicate of a page that Google recognizes as canonical, and it correctly points to that canonical page.  Google Search Console will tell you that there is nothing to do with this message.  If you want to eliminate the same page being accessible through more than one URL, you can look at your content and see if there is a way to consolidate it.

Definition: A canonical URL is a piece of code that you can add to a duplicate page. This tells the bots which page is the “master key” and should be featured instead of the duplicate page.

Duplicate Without User Selected Canonical

This message is telling you that the page has a duplicate and neither is marked as canonical. In this instance, you can fix this message by telling Google which version of a page you would like to be indexed by using a canonical tag.

Duplicate Non-HTML Page

This message usually indicates that Google has found a related PDF that contains the same information as the HTML page, so they have chosen to index the HTML version. In most cases, this is the preference, so no action needs to be taken here.

Duplicate, Google Chose Different Canonical Than User

When you see this message, it usually means that Google thinks you have chosen the wrong version of a page as canonical. Your best course of action is to ensure you have canonical tags on all duplicate pages, and that the canonicals are consistent. You may also be able to resolve this by ensuring that you are only linking to your canonical.

Duplicate, Submitted URL Not Selected as Canonical

You’ll see this message in the coverage center when a URL is one of a set of duplicate URLs and none of them have been marked as canonical. When this happens, Google is telling you that they think another page is a better choice for canonical and they have chosen to index another URL. If you disagree with Google’s decision, set your preferred version as the canonical and ensure that internal links point to that version.

Discovered – Currently Not Indexed

This means that the page was found by Google, but it has yet to be crawled.

This happens when Google finds content that it does not feel is important enough to spend time crawling and indexing.  If you see priority pages that you want to receive organic traffic, consider how you can improve internal linking on your site.  Having more links from related content within your site can help communicate that information on the page is valuable and worthy of crawling and indexation.  Another option is to reach out to relevant websites to see if they will link to the page.  Google emphasizes content that is being linked to from other websites.

Page Removed Because of Legal Complaint

If you see this message, it means that the page was removed from indexation because of a legal complaint. If you receive such a message, remove any copyrighted content and ensure that your website has not been hacked.

Page With Redirect

You will often see this message in the coverage report if you have linked to an old version of a URL. Your best bet in this instance is to update any internal links so search engines and human users don’t have to experience a redirect to engage with your content.

Submitted URL Dropped

When you see this message Google is letting you know that your page was dropped from the index for a non-specific reason. This can be frustrating because a specific cause is not given so you don’t know where to take action. Often content updates or upgrades can help with the re-indexation of the page.

What Does This Mean for You?

Indexation is important for the overall visibility of your website, but it need not be a source of frustration or anxiety. Having a better understanding of these messages in the Google Search Console coverage report will empower you to improve your indexation where necessary. While you want all of your primary product and service pages indexed, it’s also important to remember that 100% indexation is not always necessary or possible.

Need a partner in indexation and all things search engine optimization?  Our team would love to discuss your specific opportunities for improvement.  Contact us today!