As Google becomes more aware of the sustainability of crawling and indexing, Google may crawl pages less frequently.
This topic was discussed by Google’s Search Relationships team, consisting of John Mueller, Martin Splitt, and Gary Illyes.
On the latest episode of the Search Off the Record podcast, they come together to discuss what to expect from Google in 2022 and beyond.
Topics they discussed included crawling and indexing, which SEO professionals and website owners said they had seen less of in the past year.
That will be a focus for Google this year as it aims to make crawling more sustainable by saving computing resources.
What this means for your website and its performance in search results.
Sustainability of crawling and indexing
Since Googlebot crawls and indexes virtually, you might not expect this to have an environmental impact.
Illyes drew attention when he said that computing in general is unsustainable:
“…I mean, computing in general is not really sustainable. For example, if you think about Bitcoin, Bitcoin mining has a real impact on the environment that you can actually measure, especially when electricity comes from When coal-fired power plants or other less sustainable power plants.
We’re carbon free because I don’t even know if it was 2007 or 2009, but that doesn’t mean we can’t reduce our impact on the environment further. Crawling is one of the early things we can chop off some low hanging fruit. “
In this case, the low-hanging fruit is unnecessary web crawling. For example, crawl pages that haven’t had any recent updates.
How will Google make crawling more sustainable?
Illyes went on to explain that web crawling can be reduced by reducing refresh crawl.
There are two types of Googlebot crawls: crawl to discover new content and crawl to refresh existing content.
Google is considering scaling back crawls to refresh content.
Illyes continued:
“…one of the things we do, and we probably don’t need to do that much, is refresh the crawl. That means, once we’ve found a document, a URL, and we’re going to crawl it, and then, Eventually, we’ll go back and revisit that URL. That’s a refresh crawl.
And then every time we come back to that URL, there’s always a refresh. Now, how often do we need to return that URL? “
He goes on to give examples that some websites require heavy refresh crawls for some parts of the site, but not others.
A site like The Wall Street Journal constantly updates its homepage with new content, so it’s worth doing a lot of refresh crawls.
However, WSJ is unlikely to update its About page frequently, so Google doesn’t need to continue to refresh crawl these types of pages.
“So you don’t have to go back there that much. Usually, we don’t get a good estimate of this, and we definitely have room for improvement in terms of refreshing our crawlers. Because sometimes it seems wasteful that we hit the same URL over and over.
For example, sometimes we visit 404 pages for no reason. All of these are basically things we can improve and then further reduce our footprint. “
If Google is going to reduce refresh crawls, it’s not 100% sure, and that’s the impact it could have on your site.
What does a lower crawl rate mean for your website?
There is an argument that a high crawl rate is a positive SEO signal, even if you don’t update your content as often as Google crawls it.
That’s a misconception, Illyes said, because content doesn’t necessarily rank better because it’s crawled more.
Mueller:
“So I guess it’s also a misconception by people that the more a page is crawled, the higher it will rank. Is this a misconception, is this correct, or is it really true?”
Elis:
“This is a misunderstanding.”
Mueller:
“Ok, so if something hasn’t actually changed, there’s no need to try to force a recrawl. It won’t rank better.”
Again, Google hasn’t confirmed that refresh crawls will be reduced, but it’s an idea the team is actively considering.
If Google sticks to this idea, it’s not a bad thing for your website. More crawls don’t mean better rankings.
Also, the idea is to know which pages need Refresh the crawl, which pages are not. This means that pages you change more frequently will continue to refresh and update in search results.
For more details on how Google plans to make this happen, listen to the full discussion in the podcast below (starting at 2:40):
Featured image: Alena Veasey/Shutterstock
