Google’s John Mueller received erroneous feedback on how Search Console validates rich results. Due to an error in the CDN hosting the image processing the request for a non-existent robots.txt, Google will remove the image from the rich search results. The bug found was in how the search console and Google’s rich results testing failed to alert the publisher of the bug and then successfully validate the structured data.
A bug in the programming context is when a software program behaves in an unexpected way. A bug isn’t always a problem in the coding, but in this case it could be an unanticipated problem, which in turn can lead to unexpected results, like this one.
The poster who asked this question tried to use Google’s tool to diagnose why their rich media search results were disappearing, and was surprised to find that they were not useful for this particular error.
While this issue affects recipe enrichment image previews in Google Recipe enrichment results, this issue may also be an issue in other contexts.
So it’s better to be aware of this issue as it may surface in other ways.
Recipe rich result image preview disappeared
The person who asked the question provided context for what happened.
He recounts what happened:
“I would say we’ve hit a tiger trap in terms of rich recipe results.
We have hundreds of thousands of indexed recipes and a huge amount of traffic from the recipe library.
Then…after a while, it stopped.
All the metadata is checked out and the google search console says…it’s all rich recipe content and everything is fine to display.
We finally noticed that in the preview, when you preview the result, the image is missing.
Google seems to have changed, if robots.txt is required to retrieve images, nothing we see in the tool actually says anything is invalid.
So it’s kind of awkward when you check something and say “is this a valid rich recipe result?” and it says yes, it’s great, it’s absolutely great, we have all the metadata.
You checked all URLs and all images are correct, but it turns out that you have a new requirement for robots.txt. “
John Mueller asked:
“You mean you have to have a robots.txt?”
The questioner replied:
“We found that if you request robots.txt from our CDN, it will give you 500.
When we put the robots.txt in there, the preview started showing up correctly immediately.
I think this involves scraping and putting it on a static site.
So we were operational and we found that adding robots.txt did the job. “
John Mueller nodded and said:
“OK.
So from our point of view, the robots.txt file is not needed. But it must have a correct result code.
So if you don’t have on, it should return 404.
If you do have one, we can obviously read it.
However, if you return a server error for the robots.txt file, our system will assume there may be a problem with the server and we won’t crawl.
It’s been that way since the beginning.
But these kinds of issues, especially when you’re on a CDN and it’s on a separate hostname, are sometimes really hard to spot.
I’ve imaged the rich-results test, which, at least as far as I can tell, focuses on what’s on the HTML page.
So the JSON-LD markup you have there may not check if the image is actually fetchable.
Then, if they can’t be fetched, then of course, we can’t use them in the carousel either.
So this might be something we need to figure out how to highlight better. “
500 error response to CDN Robots.txt can cause problems
This is one of those SEO issues that prevents hard-to-diagnose but can lead to many negative issues, as the questioner pointed out.
Typically, a crawling of robots.txt that does not exist results in a server response code of 404, which means that robots.txt does not exist.
Therefore, if a request to the robots.txt file produces a 500 response code, it indicates that something is misconfigured on the server or CMS.
A short-term solution is to upload the robots.txt file.
But it might be a good idea to dig into the CMS or server to check for potential problems.
500 response code extracted by Robots.txt
Negative impact on recipe rich result preview due to CDN returning a 500 error response can be a rare issue.
The 500 server error response code sometimes occurs when there is something unexpected or missing in the code, and the server responds by ending code processing and throwing a 500 response code.
For example, if you edit a PHP file and forget to indicate the end of a piece of code, this can cause the server to give up processing the code and throw a 500 response.
Whatever the reason for the error response when Google tries to get robots.txt, when this happens to you, keep in mind that it’s a good question.
Citation
CDN errors for image and recipe rich results
Watch the 51:45 minute mark
