In the past few weeks, there have been discussions on social media that Googlebot has drastically reduced its crawl rate. For example, the founder of the web crawl analysis service posted a graph on Twitter showing how Google’s crawl activity has declined since November 11, 2021.
Although the slowdown in indexing will not affect all websites, many people on Twitter and Reddit agree that Google has made some changes in indexing, as evidenced by screenshots of Googlebot activity.
Evidence to reduce crawling
Anecdotal evidence of abnormal Google crawling has piled up on social media. The problem with social media is that people can literally make any observations about Google and almost guarantee their consent.
The anecdote is interesting, but not as good as Data-supported observations, This is the content that recently appeared on Twitter.
Founder of crawler and log analysis service Seolyzer (@Seolyzer_io) Released a Google crawling behavior chart, showing that crawling activity has dropped sharply since November 11.
advertise
Keep reading below
He posted:
“Googlebot is on strike! Since November 11th at 6pm (Greenwich Mean Time), Googlebot has drastically reduced its crawling activity on many large websites.”
🔥 Googlebot is on strike! 😱 Since 6pm (GMT) on November 11th, Googlebot has drastically reduced its crawling activity on many large websites. Are you worried? It needs a thread! 🔽 pic.twitter.com/ugLmCZbC1O
-Olivier @ Seolyzer.io 🚀 (@Seolyzer_io) November 15, 2021
304 server response code and Googlebot crawling
Some people noticed that Googlebot suddenly stopped crawling pages that provided a 304 server response code.
The 304 response code means 304 (unmodified).
When the browser (or Googlebot) makes a conditional request to the page, the response code is generated by the server.
This means that the browser (or Googlebot) will tell the server that it has saved a webpage in the cache, so don’t bother it unless the page has been updated (modified).
The following is the definition of the 304 (unmodified) server response code HTTP Working Group:
“The 304 (unmodified) status code indicates that a conditional GET or HEAD request has been received. If it is not because the condition is evaluated as false, it will result in a 200 (OK) response.
In other words, the server does not need to transmit a representation of the target resource, because the request indicates that the client already has a valid representation. Therefore, the server is redirecting the client to use the stored representation as if it were the payload of the 200 (OK) response. “
advertise
Keep reading below
304 response will reduce Googlebot crawling?
One person confirmed (in French) on Twitter that on several AMP-enabled websites he monitored, pages that responded to 304 responses had dropped.
I have also confirmed in the search consoles of several websites that the 304 on November 12 has dropped significantly
-Erwan Le Tallec (@eletallec) November 15, 2021
The person who posted the original tweet replied with a graph showing that Google almost stopped crawling pages that responded to the 304 server response code:
Theory 2: 304s pic.twitter.com/KQ2k1pgteS
-Olivier @ Seolyzer.io 🚀 (@Seolyzer_io) November 15, 2021
Others have noticed a similar problem, that is, the crawl rate of pages that provide 304 responses is greatly reduced:
Lol I saw this in this WE, before creating a thread, I was looking for information/announcement that can be explained, but the explanation is obviously amp and 304
-Raph Duce (@RaphSEO) November 15, 2021
Another person noticed that the crawl on the travel page decreased, but the crawl on the e-commerce page increased:
This pattern is only seen on Croatia’s tourism and tourism portals, and the e-commerce vertical is very good (even after a few nights of November 11th, the amount of crawling increased significantly)
— Marko Cvijic (@MarkoCvijic) November 15, 2021
Many others are sharing screenshots of the analytics and search console:
@John Mu I think there are more routing issues with Google crawling again. Local Nginx server and S3 header, same problem. Can you tell us something about it? Maybe a problem related to Cloudflare? https://t.co/c8eV9C4pxg @Seolyzer_io pic.twitter.com/mG1Iqb30UR
-Carlos Redondo (@carlosredondo) November 15, 2021
More data:
I extracted some data, centered on 21-11-11 19:39 (Paris time).
Since then, some of the crawled IPs verified by Google have become completely dark. pic.twitter.com/FcqeMXuJPv-Baptiste M. (@bactisme) November 15, 2021
304 response code should not change crawling
Google official developer help page Documents crawled by Googlebot It is stated that the 304 response code should not affect crawling.
The following are suggestions from Google’s official documentation:
“Googlebot sends a signal to the indexing pipeline that the content is the same as it was last crawled.
The index pipeline may recalculate the URL signal, otherwise the status code has no effect on the index. “
advertise
Keep reading below
Is it possible that Google has changed (permanently or temporarily) and the developer page is out of date?
Cookie Consent Theory
The 304 server response theory is one of many theories and solutions explaining why Googlebot may not index web pages.
One person said on Twitter that Google added an index after deleting the cookie consent field.
Does Google stop crawling and indexing new pages? I encountered the same problem and deleted the cookie consent bar (Cookiepro) for testing. Guess what-the problem is solved. @John Mu – Any ideas why Google may not use cookies to agree to pop-up windows to crawl and index new pages?
-Dennis Sievers (@resiever) November 16, 2021
Why does the cookie response bar cause Google not to index web pages? Did the cookie consent bar trigger a 304 response, preventing Google from indexing the page?
Reducing Googlebot crawls discussed on Reddit
Reddit also discussed the reduction of Googlebot crawling.
A Redditor described how articles on their successful websites in the past were indexed within 10 minutes after being submitted through Google Search Console.
They say that only half of the new articles have been indexed recently.
advertise
Keep reading below
But according to the situation in November This Reddit post:
“For whatever reason, less than half of our new articles are now indexed, even if I submit them manually after publishing.”
Other redditors shared similar experiences:
“A lot of people are experiencing similar things now… Something seems to have happened at Google.”
“Something went wrong when Google indexed new posts…”
“My website is 17 years old… Suddenly, it took a few weeks for the latest article to be included.”
Google says there are no problems
John Mueller of Google Reply to questions on Reddit:
“At the moment, I don’t think there is any problem with the way Google indexes content. However, I do see that we are critical of the indexed content because any search engine should do this.”
Is Google testing a new crawl mode?
Bing announced in October an open source indexing agreement called Index now Its goal is to reduce the frequency of crawling web pages by crawlers, so as to reduce the energy used in data centers for crawling and servers to provide web pages. The new agreement benefits publishers because it speeds up the process of notifying search engines when a page is updated or created, thereby speeding up the indexing of high-quality web pages.
advertise
Keep reading below
November Google announced It will test the new IndexNow indexing protocol to see if it is beneficial.
Saving energy and reducing carbon footprint is one of our most important issues today. Could it be that Google is improving its methods to reduce crawling, and will not fundamentally change to a new protocol?
Has Google reduced web crawling?
Some people claim that Google has completely stopped indexing, but this is incorrect. However, there has been a lot of discussion on data-backed social media to support the Googlebot indexing model has changed.
