A technical SEO published a case study showing how he solved a curious problem Crawled and not currently indexed Problems on his website. Although the solution he found may not be universal to others who encounter this problem, his method of identifying and solving the problem provides a useful walkthrough for solving technical SEO problems.
What happened to his website index is really strange. But his solution is simple and reasonable.
I found a description of this problem on a tweet by Adam Gent (@Adoubleagent)
A small blog post about the technical SEO issues I encountered on my small website.
A strange case of standardization -> https://t.co/pC2QAYLjq9
TL; DR-Google’s standardization can be very wrong, which will affect SEO traffic.
-Adam Gent (@Adoubleagent) November 3, 2021
advertise
Keep reading below
Crawled-not currently indexed
In Facebook, Twitter, and even John Mueller’s office hours gatherings, there are many anecdotal reports about “currently unindexed crawls.”
In a recent office hour hangout, someone asked why Google Search Console (GSC) displayed “crawl not indexed”, but when you clicked on them, the results were indexed. John Mueller answers This is just a lag between reports.
Meet in another office hour John Mueller points out It is perfectly normal for a website to have many pages that are not indexed.
He pointed out:
“…If you have a smaller website and you find that a large part of your pages are not indexed, then I will take a step back and try to reconsider the overall quality of the website instead of focusing too much on technical issues. page.
Another thing to remember about indexing is that it’s perfectly normal for us not to index everything on the site.
Over time, when you like 200 pages on the site and we index 180 of them, this percentage will decrease. “
advertise
Keep reading below
Although these two are good reasons to explain why some people have the Crawled Not Indexed problem, this is not the reason Adam Gent discovered.
Adam Gent found a completely different problem, it seems to be Google’s own algorithm problem. There is no problem with the site itself, the problem lies in Google’s indexing.
Why crawl-currently not indexed
Adam checked the GSC index coverage report and found that Google was crawling and indexing his feeds as if they were HTML pages.
He randomly extracted words from these pages and created a site: using these words to search and found that the content of the feed page was indeed indexed.
To make matters worse, Google has apparently standardized the content of RSS feeds on actual web pages, explaining why the actual web pages were crawled but not indexed.
RSS feed is generated by WordPress
One strange thing about this situation is that when you view the feed page, it appears like a web page, rather than the way XML files are usually presented.
Screenshot of RSS feed cache
I may be wrong, but this does not look like a normal RSS feed. It looks like an HTML page.
advertise
Keep reading below
Although the underlying code is indeed XML, this is not what most feeds usually look like.
Could this be the reason why Google chose to normalize the feed?
It’s hard to understand how this happens because there are too many signals such as internal links, which under normal circumstances will cause Google to prefer HTML pages as the norm.
How Adam solved the problem
After Adam figured out what was happening, he deleted the feed pages generated by WordPress, submitted the feed URL for crawling, and then processed the 404 on the page.
After these pages were removed from the index, he next submitted the correct URL to Google and resolved the issue within a few days.
advertise
Keep reading below
What caused the problem?
Adam wrote that the problem seems to be on Google’s side.
I asked around and someone told me that apparently Google started indexing feeds a few years ago, but he thought the problem had been resolved.
I am not an expert in XML, but it seems unusual that the feed resembles an HTML page instead of a normal XML layout (no HTML style display).
The feed does not look normal, so it seems that what makes it look like may be a potential cause.
In any case, if you encounter the “crawl is not currently indexed” issue, this is another thing to check in case it also happens to you.
advertise
Keep reading below
Citation
Read the original post that solved the problem:
