A data science method for optimizing internal link structure

If you care about whether your website pages have sufficient authority to rank for their target keywords, then optimizing internal links is important. By internal links, we mean that the pages on your website receive links from other pages.

This is important because it is the basis for Google and other searches to calculate the importance of pages relative to other pages on your website.

It also affects the likelihood that users will find content on your website.Content discovery is the foundation Google page rank algorithm.

Today, we are exploring a data-driven method to improve the internal links of the website to achieve more effective technical website SEO. That is to ensure that the distribution of internal domain permissions is optimized according to the site structure.

Use data science to improve internal link structure

Our data-driven approach will only focus on optimizing one aspect of the internal link architecture, which is to model the distribution of internal links according to the depth of the site, and then locate pages that lack deep links to specific sites.

We first import the library and data, and clean up the column names before previewing them:

import pandas as pd
import numpy as np
site_name="ON24"
site_filename="on24"
website="www.on24.com"

# import Crawl Data
crawl_data = pd.read_csv('data/'+ site_filename + '_crawl.csv')
crawl_data.columns = crawl_data.columns.str.replace(' ','_')
crawl_data.columns = crawl_data.columns.str.replace('.','')
crawl_data.columns = crawl_data.columns.str.replace('(','')
crawl_data.columns = crawl_data.columns.str.replace(')','')
crawl_data.columns = map(str.lower, crawl_data.columns)
print(crawl_data.shape)
print(crawl_data.dtypes)
Crawl_data

(8611, 104)

url                          object
base_url                     object
crawl_depth                  object
crawl_status                 object
host                         object
                             ...   
redirect_type                object
redirect_url                 object
redirect_url_status          object
redirect_url_status_code     object
unnamed:_103                float64
Length: 104, dtype: object

The image above shows a preview of the data imported from the Sitebulb desktop crawler application. There are more than 8,000 lines, and not all lines are unique to the domain because it also includes resource URLs and external outbound link URLs.

We have more than 100 columns that are redundant for requirements, so we need to select some columns.

However, before we start, we want to quickly understand how many site levels there are:

crawl_depth
0             1
1            70
10            5
11            1
12            1
13            2
14            1
2           303
3           378
4           347
5           253
6           194
7            96
8            33
9            19
Not Set    2351
dtype: int64

So as can be seen from the above, there are 14 site levels, most of which are not in the site architecture, but in the XML site map.

You may notice that Pandas (a Python package for processing data) sorts the site level numerically.

This is because the site level is a string instead of a number at this stage. This will be adjusted in future code because it will affect data visualization (“viz”).

Now we will filter the rows and select the columns.

# Filter for redirected and live links

redir_live_urls = crawl_data[['url', 'crawl_depth', 'http_status_code', 'indexable_status', 'no_internal_links_to_url', 'host', 'title']]
redir_live_urls = redir_live_urls.loc[redir_live_urls.http_status_code.str.startswith(('2'), na=False)]
redir_live_urls['crawl_depth'] = redir_live_urls['crawl_depth'].astype('category')
redir_live_urls['crawl_depth'] = redir_live_urls['crawl_depth'].cat.reorder_categories(['0', '1', '2', '3', '4',
                                                                                 '5', '6', '7', '8', '9',
                                                                                        '10', '11', '12', '13', '14',
                                                                                        'Not Set',
                                                                                       ])
redir_live_urls = redir_live_urls.loc[redir_live_urls.host == website]
del redir_live_urls['host']
print(redir_live_urls.shape)
Redir_live_urls

(4055, 6)

Andreas Voniatis, November 2021

By filtering the rows of indexable URLs and selecting the relevant columns, we now have a more streamlined data frame (think the spreadsheet tab of the Pandas version).

Explore the distribution of internal links

Now we are ready to visualize the data and understand how to Internal link Distributed in the overall and site depth.

from plotnine import *
import matplotlib.pyplot as plt
pd.set_option('display.max_colwidth', None)
%matplotlib inline

# Distribution of internal links to URL by site level
ove_intlink_dist_plt = (ggplot(redir_live_urls, aes(x = 'no_internal_links_to_url')) + 
                    geom_histogram(fill="blue", alpha = 0.6, bins = 7) +
                    labs(y = '# Internal Links to URL') + 
                    theme_classic() +            
                    theme(legend_position = 'none')
                   )

ove_intlink_dist_plt

Andreas Voniatis, November 2021

From the above we can see that most pages do not have links, so improving internal links will be an important opportunity Improve search engine optimization here.

Let’s get some statistics at the site level.

crawl_depth
0             1
1            70
10            5
11            1
12            1
13            2
14            1
2           303
3           378
4           347
5           253
6           194
7            96
8            33
9            19
Not Set    2351
dtype: int64

The above table shows the rough distribution of internal links by site level, including the mean and median (50% quantile).

Together with the variation within the site level (standard deviation is std), it tells us how close the page is to the average within the site level; that is, the consistency of the distribution of internal links with the average.

From the above, we can infer that, except for the home page (crawl depth 0) and first-level pages (crawl depth 1), the average site-level URL ranges from 0 to 4.

For a more intuitive method:

# Distribution of internal links to URL by site level
intlink_dist_plt = (ggplot(redir_live_urls, aes(x = 'crawl_depth', y = 'no_internal_links_to_url')) + 
                    geom_boxplot(fill="blue", alpha = 0.8) +
                    labs(y = '# Internal Links to URL', x = 'Site Level') + 
                    theme_classic() +            
                    theme(legend_position = 'none')
                   )

intlink_dist_plt.save(filename="images/1_intlink_dist_plt.png", height=5, width=5, units="in", dpi=1000)
intlink_dist_plt

Andreas Voniatis, November 2021

The image above confirms our previous comment that the homepage and directly linked pages get most of the links.

In terms of these scales, we don’t have much opinion on the lower-level distribution. We will modify it by taking the logarithm of the y-axis:

# Distribution of internal links to URL by site level
from mizani.formatters import comma_format

intlink_dist_plt = (ggplot(redir_live_urls, aes(x = 'crawl_depth', y = 'no_internal_links_to_url')) + 
                    geom_boxplot(fill="blue", alpha = 0.8) +
                    labs(y = '# Internal Links to URL', x = 'Site Level') + 
                    scale_y_log10(labels = comma_format()) + 
                    theme_classic() +            
                    theme(legend_position = 'none')
                   )

intlink_dist_plt.save(filename="images/1_log_intlink_dist_plt.png", height=5, width=5, units="in", dpi=1000)
intlink_dist_plt

Andreas Voniatis, November 2021

The above shows the same link distribution as the logarithmic view, which helps us confirm the lower-level distribution average. This is easier to visualize.

Given the difference between the first two site levels and the rest, this indicates a skew in the distribution.

Therefore, I will take the logarithm of internal links, which will help normalize the distribution.

Now that we have a standardized number of links, we will visualize it:

# Distribution of internal links to URL by site level
intlink_dist_plt = (ggplot(redir_live_urls, aes(x = 'crawl_depth', y = 'log_intlinks')) + 
                    geom_boxplot(fill="blue", alpha = 0.8) +
                    labs(y = '# Log Internal Links to URL', x = 'Site Level') + 
                    #scale_y_log10(labels = comma_format()) + 
                    theme_classic() +            
                    theme(legend_position = 'none')
                   )

intlink_dist_plt

Andreas Voniatis, November 2021

As you can see from the above, the distribution looks less skewed because the box (interquartile range) has a more gradual step change from site level to site level.

This allows us to analyze the data well, and then diagnose which URLs are not optimized from the perspective of internal links.

Quantification problem

The following code will calculate the lower 35 quantiles (data science term for percentiles) of the depth of each site.

# internal links in under/over indexing at site level
# count of URLs under indexed for internal link counts

quantiled_intlinks = redir_live_urls.groupby('crawl_depth').agg({'log_intlinks': 
                                                                 [quantile_lower]}).reset_index()
quantiled_intlinks = quantiled_intlinks.rename(columns = {'crawl_depth_': 'crawl_depth', 
                                                          'log_intlinks_quantile_lower': 'sd_intlink_lowqua'})
quantiled_intlinks

Andreas Voniatis, November 2021

The figure above shows the calculation result. These numbers are meaningless to SEO practitioners at this stage, because they are arbitrary, and the purpose is to provide a cut-off point for URLs with insufficient links at each site level.

Now that we have the tables, we will merge them with the main data collection to determine if the line-by-line URL is insufficiently linked.

# join quantiles to main df and then count
redir_live_urls_underidx = redir_live_urls.merge(quantiled_intlinks, on = 'crawl_depth', how = 'left')

redir_live_urls_underidx['sd_int_uidx'] = redir_live_urls_underidx.apply(sd_intlinkscount_underover, axis=1)
redir_live_urls_underidx['sd_int_uidx'] = np.where(redir_live_urls_underidx['crawl_depth'] == 'Not Set', 1,
                                                   redir_live_urls_underidx['sd_int_uidx'])

redir_live_urls_underidx

Now we have a data frame, and each URL is marked as 1 under the “sd_int_uidx” column.

This allows us to summarize the number of under-linked site pages by site depth:

# Summarise int_udx by site level
intlinks_agged = redir_live_urls_underidx.groupby('crawl_depth').agg({'sd_int_uidx': ['sum', 'count']}).reset_index()
intlinks_agged = intlinks_agged.rename(columns = {'crawl_depth_': 'crawl_depth'})
intlinks_agged['sd_uidx_prop'] = intlinks_agged.sd_int_uidx_sum / intlinks_agged.sd_int_uidx_count * 100
print(intlinks_agged)

  crawl_depth  sd_int_uidx_sum  sd_int_uidx_count  sd_uidx_prop
0            0                0                  1      0.000000
1            1               41                 70     58.571429
2            2               66                303     21.782178
3            3              110                378     29.100529
4            4              109                347     31.412104
5            5               68                253     26.877470
6            6               63                194     32.474227
7            7                9                 96      9.375000
8            8                6                 33     18.181818
9            9                6                 19     31.578947
10          10                0                  5      0.000000
11          11                0                  1      0.000000
12          12                0                  1      0.000000
13          13                0                  2      0.000000
14          14                0                  1      0.000000
15     Not Set             2351               2351    100.000000

We now see that although the number of links per URL on the site depth 1 page is higher than average, there are still 41 pages with insufficient links.

To be more intuitive:

# plot the table
depth_uidx_plt = (ggplot(intlinks_agged, aes(x = 'crawl_depth', y = 'sd_int_uidx_sum')) + 
                    geom_bar(stat="identity", fill="blue", alpha = 0.8) +
                    labs(y = '# Under Linked URLs', x = 'Site Level') + 
                    scale_y_log10() + 
                    theme_classic() +            
                    theme(legend_position = 'none')
                   )

depth_uidx_plt.save(filename="images/1_depth_uidx_plt.png", height=5, width=5, units="in", dpi=1000)
depth_uidx_plt

Andreas Voniatis, November 2021

Apart from XML sitemap URLs, the distribution of low link URLs looks normal, as shown in a nearly bell shape. Most of the URLs under the link are at site levels 3 and 4.

Export a list of unlinked URLs

Now that we have mastered the URL of the link at the site level, we can export the data and propose creative solutions to bridge the gap in site depth, as shown below.

# data dump of under performing backlinks
underlinked_urls = redir_live_urls_underidx.loc[redir_live_urls_underidx.sd_int_uidx == 1]
underlinked_urls = underlinked_urls.sort_values(['crawl_depth', 'no_internal_links_to_url'])
underlinked_urls.to_csv('exports/underlinked_urls.csv')
underlinked_urls

Andreas Voniatis, November 2021

Other data science techniques for internal linking

Before exploring how internal links are distributed throughout the site at the site level, we briefly introduced the motivation for improving internal links on the site.

Then, before exporting the results for recommendation, we continue to quantify the extent of the lack of link problem digitally and visually.

Of course, the site level is only one aspect of internal links that can be statistically explored and analyzed.

Other areas where data science techniques can be applied to internal links include, but are obviously not limited to:

• Remote page level permissions.
• Anchor text relevance.
• Search intent.
• Search user journeys.

What do you want to see?

Please comment below.

More resources:

Featured image: Shutterstock/Optimarc