Google indexing is different to ranking favourably for keywords which requires months of effort to achieve. Being indexed on Google means your website is added to Google’s database. It is the first step towards building your Google rank. If a website is not indexed by Google, it will not show up on search results.

How to find if your website is indexed properly by Google?

Searching on Google with the site command will show all pages of your website that are indexed by Google. See example  below:

site: www.yoursitename.com

See image below:

If you believe a significant number of your pages are not listed, then your website may be suffering from one of the indexing problems below.

How to Identify and Fix Crawling Problems

The easiest way to determine crawling issues is to log in to your Google Webmaster Tools dashboard. It is a great free tool that shows problems associated with your website including indexing or crawling errors.

See example image below:

Main Crawling Errors That prevent WebPages from getting indexed:

Typically these kinds of issues are caused by one or more of the following reasons:

1. Wrong HTTP Status Code

Whenver a webpage is accessed online, the server returns an internal status code for the page e.g. 200 for success or 404 for page not found. Incorrect HTTP status code is one of the most common reasons for crawling or indexing errors.

2. Problems with your Robots.txt

Robots.txt is a simple text file that resides in the root folder of your website on server. It is used to inform search engine crawlers the areas of the site that should be indexed on search engines and the areas that should not.

Incorrect use can block access to relevant parts of your website.

3.  Problems with htaccess File

Htaccess is an invisible file on your server that is used for a number of configuration updates. A poorly configured htaccess file can cause many unexpected issues including prevent search engines from indexing your website pages.

4.  Meta Tags

There are special HTML Meta tags such as NOINDEX and NOFOLLOW that are specifically designed to prevent search engines from indexing a webpage. The feature exists as it can be useful in some instances. However incorrect use will prevent your website pages from appearing on search engine results.

You can easily look at the source of the page concerned, to see if this tag is present. It is typically at the top in the head section of your code.

<META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”>

5. Problems with Your Sitemap

Webmasters often use a special XML sitemap to make sure all pages are indexed by search engines, and not just the homepage. If there are errors with your sitemap it can affect how your website is indexed by search engines.

On Google, you can check your Webmaster Tools dashboard to see if a sitemap has been added or if there are problems with it.

6. You don’t have enough Pagerank

Matt Cutts revealed in an interview with Eric Enge that the number of pages Google crawls is roughly proportional to your pagerank.

7. Connectivity or Server Issues

If your website or server is down for any reason, Google cannot crawl or index your site.

8. Framed Forwarding

There are several methods to forward a website to another URL if you need to do it. The correct way is to use a 301 redirect command in your HTACCESS file. However many websites incorrectly use frame based forwarding instead. This can prevent a website from being indexed by search engines.

How to check if your website uses Frames

If you are not technical, a quick way to check if your website uses frames is to look at your address bar. If the address bar always displays a single URL i.e. “www.yourwebsitename.com” on all pages, then your website most likely uses frames.

9.  Search Engine Penalty

If your website has been associated with malpractices in the past, it can seriously affect your chances of being indexing on search engines. This includes spam, malware, copying content from other sites, or other malpractices.

It is also possible that the domain name was previously owned by a different person who may have used the site in such ways. If this is the case you can file a reconsideration request from Google Webmaster Tools dashboard.

CONCLUSION:

The crawl errors discussed above are the most common reasons for problems with Google indexing. If your website suffers from Google indexing problems, you should address them immediately if you want to build your rank and improve your visibility on search engines.