« Google AdWords answering questions | Main | Google in Tamil – News, Orkut, Blogger and Search »
Infinity Space - Google Crawling and Indexing issue
By Find2k | August 6, 2008
As more and more websites are doing online marketing activities, the chances of more SEO and online branding will occur for most of the websites.Google started realizing some conflicts/ issues they are having in the crawling and indexing of their crawler “Googlebot”.
Sometimes, when the crawler crawls the web goes into an “infinite space” due to a large number of links, these large numbers of links are those that usually provide little or no new content for Googlebot to index and if this happens on your site, crawling those URLs may use unnecessary bandwidth, and could result in Googlebot failing to completely index the real content on your site.
So do you see danger here, as googlebot won’t be able to read the content of your site, so what you can do, as how you will know what to do? Well that’s the reason why Google have started notifying site owners when they discover this problem on the web sites.
Like most of the message they send, you will find these messages in Webmaster Tools in the Message Center. You’ll probably want to know right away if Googlebot has this problem - or other problems - crawling your sites. So verify your site with Webmaster Tools, and check the Message Center every now and then and keep your site away from the “infinite space”.
A common scenario is websites which provide for filtering a set of search results in many ways. A shopping site might allow for finding clothing items by filtering on category, price, color, brand, style, etc.
The number of possible combinations of filters can grow exponentially. This can produce thousands of URLs, all finding some subset of the items sold. This may be convenient for your users, but is not so helpful for the Googlebot, which just wants to find everything - once!
Topics: Google, Internet Marketing, New Updates on Google, Search Marketing |