Want your product listings to rank higher on search engines?
Feel like your success is as limited as your crawl budget?
Effectively ranking thousands of products, hundreds of categories, and millions of links requires a level of organization that can sometimes feel out of reach.
This is especially true when the performance of your ecommerce site depends on a limited crawl budget or an incoherent team.
For large e-commerce sites, this is a huge challenge.
In this article, you’ll find proven solutions to some of the most persistent technical SEO problems plaguing sites like yours. You’ll learn how to address issues related to crawl budget, site architecture, internal linking, and more that are hindering your site’s performance.
let’s start.
1. Crawl budgets are often too limited to provide actionable insights
Growing your e-commerce business is great, but it can result in tons of pages and a cluttered, outdated website structure.
Your company’s incredible growth can lead to:
- Broad SEO crawl budget needs.
- long crawling process.
- Outdated content that is easy to miss, such as orphaned pages and zombie pages that no longer need to be crawled or indexed, leads to a waste of high crawl budgets.
- Incomprehensible reports filled millions of pages with repeated basic technical errors.
- Incomplete and fragmented crawl data, or partial crawls.
Try to solve SEO problems using Partial crawling is not a good idea; you won’t be able to find all the mistakes, leading you to make SEO decisions that could do more harm than good.
Whether your crawl budget constraints come from site size or desktop-based crawlers, you need a solution that gives you unlimited visibility and understanding of your entire site.
Solution: Use raw logs instead of scraping reports
To overcome the slow and limited crawl budget, we recommend using raw logs instead of crawl reports.
Raw logs enable you to:
- Monitor crawling, indexing, and detailed content analytics at a more affordable price.
- Know which pages are impacting your crawl budget and optimize accordingly.
- Eliminate critical bugs immediately after product updates.
- Allows you to fix problems before Google bots find them.
- Quickly identify pages with 2XX, 3XX, 4XX, and 5XX status codes.
Using a raw log tool can also give you an accurate picture of your website’s SEO efficiency.
You’ll be able to pull reports showing the number of pages in your site structure, pages that get visits from search bots, and pages that get impressions in the SERPs.
This gives you a clearer picture of where structural and crawling problems occur at any depth.
Screenshot of JetOctopus, November 2021For example, we can see that there are over 4 million pages in the site structure above.
Search bots only visited 725,161.
Only 29,277 of these pages were ranked and received impressions.
The 24,189,025 pages visited by search bots are not even part of the site structure.
What a rare opportunity!
How to Use Raw Logs to Find and Fix SEO Crawl Issues Faster
Implement an unlimited SEO analysis tool that can crawl complete websites of any size and structure.
JetOctopus is extremely fast, crawling up to 250 pages per second or 50,000 pages in 5 minutes to help you understand how your crawl budget is being impacted.
Simply:
- Create an account at JetOctopus.
- Visit the Impact section.
- Evaluate your crawl rate and missed pages.
In seconds, you can measure the percentage of SEO effective pages and know how to improve the situation.
Screenshot of JetOctopus, November 2021Our log analyzer tracks crawl budgets, bots and orphaned pages, accessibility errors, crawl defect areas, bot behavior (by distance from index, content size, inbound links, most active pages, etc.).
With its efficient visual representation, you can improve indexability while optimizing your crawl budget.
Screenshot of JetOctopus, November 2021how?
Crawl budget optimization is at the heart of any SEO effort, especially for large sites. Here are a few points to help you get started.
Determine if your crawl budget is being wasted.
Log file analysis can help you determine why your crawl budget is wasted.
Please visit the Log File Analysis section to determine this.
Get rid of error pages.
View your site’s crawl through log file analysis for pages that may have 301, 400, or 500 errors.
Improve grabbing efficiency.
Use SEO crawl and log file data to identify differences between crawled and indexed pages. Consider the following to improve crawl efficiency.
- Make sure the GSC parameter settings are up to date.
- Check if any important pages are included as non-indexable pages. The data in the log files will help you find them.
- Add disallowed paths in robots.txt file to save crawl budget for priority pages.
- Add relevant noindex and canonical tags to indicate their importance to search bots. However, the noindex tag does not work well in the case of multimedia resources (ie video and PDF files). In this case, use robots.txt.
- Find disallowed pages that search bots are crawling.
2. Managing a large internal linking structure can be complicated
Internal linking is one of the best ways you can tell Google about what’s on your site.
When you create a link to a product from a page on your website, you give Google a clear crawl path to rank your page.
Google’s crawlers use a site’s internal link structure and anchor text to gain contextual meaning and discover other pages on the site.
Screenshot of SearchEngineJournal.com, November 2021However, creating a crawl friendly Internal link structure Difficult for large sites.
Keeping up with internally linked products that are constantly coming in and out of stock isn’t always sustainable on a large e-commerce site.
You need a way to see where dead ends occur during Google’s crawl.
Why Internal Linking Structure Matters
Google relies on internal linking to help it understand how quickly and easily visitors navigate websites.
If your homepage ranks well for a particular keyword, internal linking helps distribute PageRank to other, more focused pages throughout your site. This helps those linked pages rank higher.
Solution: Use the Interconnect Fabric Efficiency Tool to Find Crawl Dead Ends
Our Interlinking Structure Efficiency solves this problem, giving you a clear picture of your site’s internal linking health.
- On the dashboard, go to Idea -> Structural Efficiency.
- This screenshot shows the list of directories present on the site, pages in this directory, percentage of indexable pages, average number of internal links to pages in this directory, bot behavior here, SERP impressions and clicks. It clearly reflects SEO efficiency through catalog analysis and multiplied by active experimentation.
Screenshot of JetOctopus, November 2021See how our customers are doing DOM.RIA doubles Googlebot traffic By experimenting with it.
3. Solving SEO problems on JavaScript sites is difficult
JavaScript is the cornerstone of responsive website design, enabling developers to increase the interactivity and complexity of their applications. That’s why big marketplaces like Amazon and eBay use it.
However, JavaScript websites face two problems:
- Grabability: JS content limits the ability of crawlers to browse the site page by page, affecting its indexability.
- availability: Although the crawler will read a JS page, it cannot figure out what the content belongs to. Therefore, they will not be able to rank pages for related keywords.
Therefore, e-commerce webmasters cannot determine which pages are rendered and which are not.
Solution: Use an advanced crawler that treats JavaScript like Googlebot
Traditionally, SEO crawlers cannot crawl JavaScript sites. But JetOctopus is one of the most advanced crawlers with JavaScript rendering capabilities.
In JS Performance, you’ll find insights about JavaScript execution — that is, first paint, first content paint, and page load — as well as the time it takes for all JavaScript requests to complete.
It also shows JS errors.
Here’s how to make this function work for you:
- Go to JS Performance in the Crawler tab.
Screenshot of JetOctopus, November 2021- View your site as Googlebot using JavaScript. This GIF shows the process.
Screenshot of JetOctopus, November 20214. Few tools provide deep insights for large sites
Core Web Vitals and Page Speed are important technical SEO metrics to monitor. However, few tools track these page by page.
