The new sites are indexed by Google with the help of “Bots”. A Bot is piece of software from a search Engine that is built to go through every page of a site, then categorize it, and place it to the database.
Google has three well known bots:
* The Adsense bot
* The Freshbot
* The DeepCrawl.
The Adsense Bot is used for publishers who have Adsense on their sites. As soon as a new page is created, the JavaScript within the Adsense code sends a message to the Adsense bot, and it will come within few minutes to index the page so that it can serve up the most relevant ads.
The Freshbot crawls the most popular pages on your website, doesn't matter if that is one page or thousands. A typical site should expect to have a Freshbot visit every 1 to 14 days, depending on how popular those pages are. Freshbot places those crawled links into a database so that when the DeepCrawl occurs, it has a reference.
Once a month, the DeepCrawl bot visits your site and goes over all the links found by the Freshbot. This is the reason why it can take up to a month for your entire site to be indexed in Google - even with the addition of a Google Sitemap.
