Google has two ways of crawling

Google uses two types of crawling methods when browsing the web – one for discovering new content and one for refreshing existing content.

Google’s search advocate John Mueller explained this in a Google Search Center SEO office hour hangout recorded on January 7.

An SEO professional named Swyamdipta Chakraborty joined the livestream to ask Mueller a series of questions, one of which was related to how often Googlebot crawls his site.

He noted that Googlebot used to crawl his site every day when he published his site more frequently, but not as much when he published fewer articles.

Perhaps out of concern that the reduction in crawling frequency was a bad sign, he asked Mueller if that was normal.

Mueller assured him that was fine, and went on to explain the two types of crawling Googlebot does.

Learn more about how Google crawls websites in the section below.

Two types of Googlebot crawls

Reports in Search Console allow you to see how often Googlebot crawls your site, and sometimes your site may be crawled more than others.

When asked about the report, Mueller confirmed the volatility was normal and discussed two types of crawls:

“It can happen. We don’t crawl the website, we crawl individual pages of the website. And when it comes to crawling, we have roughly two types of crawling.

One is discovery crawlers, where we try to discover new pages on your site. The other is to refresh the crawler, where we update existing pages that we know about. “

Crawl frequency will vary not only across the site, but also across individual pages.

For example, if your homepage is updated more frequently than other pages, you’ll see more Googlebot activity on that page.

Mueller continued:

“So in most cases, for example, we refresh the crawl homepage, I don’t know, once a day, or every few hours, or something like that.

If we find new links on their homepage, then we also use discovery crawling to crawl those links. Because of this, you’ll always see a mix of crawl-related discoveries and refreshes. You’ll see a baseline of some crawling happening every day.

However, if we realize that individual pages rarely change, then we realize that we don’t have to crawl them all the time. “

Certain types of websites may be easier to crawl than others.

News sites that are updated multiple times a day are crawled more often than sites that are updated once a month.

Googlebot is able to recognize these patterns and adjust its crawl rate accordingly.

“For example, if you have a news site and you update every hour, then we should know we need to crawl every hour. And if it’s a news site that is updated every month, then we should know we don’t need to crawl every hour Grab once.

It’s not a sign of quality, or a sign of ranking, or anything like that. Purely from a technical point of view, we learned that we could scrape once a day or a week, and it didn’t matter. “

So don’t panic if you find that Googlebot is visiting your site more or less.

Also, don’t worry if Googlebot has recently crawled your site and updates to existing content aren’t reflected in search results.

This could be a case of Google crawling your site to discover new content rather than refreshing existing content.

If your site rarely makes changes to published content, Googlebot may crawl more content for refresh discovery.

Again, it doesn’t necessarily have to do with content quality.

Hear the full discussion below:

Featured image: Diajoti/Shutterstock