Chances are, you used a more popular tool like Ahrefs or Semrush to analyze your site’s backlinks.
These tools search the web for a list of sites linking to your site, with domain ratings and other data describing the quality of backlinks.
it’s no secret Backlinks play an important role in Google’s algorithmso it makes sense to at least understand your own website before comparing it to your competitors.
While using tools can give you insight into specific metrics, learning to analyze backlinks yourself can give you more flexibility in what you’re measuring and how it’s presented.
Although you can do most of your analysis on a spreadsheet, Python has certain advantages.
In addition to the number of rows it can handle, it also makes it easier to see statistical aspects such as distribution.
In this column, you’ll find step-by-step instructions on how to use Python to visualize basic backlink analysis and custom reports that take into account different link attributes.
Does not occupy a seat
We’ll pick a small website from the UK furniture industry as an example and do some basic analysis using Python.
So what is the value of a website’s backlinks for SEO?
The simplest, I would say quality and quantity.
Quality is subjective to experts, but certain to Google through metrics like authority and content relevance.
Before evaluating quantity, we will first evaluate link quality using available data.
Time to code.
import re
import time
import random
import pandas as pd
import numpy as np
import datetime
from datetime import timedelta
from plotnine import *
import matplotlib.pyplot as plt
from pandas.api.types import is_string_dtype
from pandas.api.types import is_numeric_dtype
import uritools
pd.set_option('display.max_colwidth', None)
%matplotlib inline
root_domain = 'johnsankey.co.uk'
hostdomain = 'www.johnsankey.co.uk'
hostname="johnsankey"
full_domain = 'https://www.johnsankey.co.uk'
target_name="John Sankey"
We import the data first and clean up the column names to make it easier to handle and faster to type for later stages.
target_ahrefs_raw = pd.read_csv( 'data/johnsankey.co.uk-refdomains-subdomains__2022-03-18_15-15-47.csv')
List comprehensions are a powerful and less intensive way to sanitize column names.
target_ahrefs_raw.columns = [col.lower() for col in target_ahrefs_raw.columns]
The list comprehension instructs Python to convert the column name of each column (‘col’) in the data frame to lowercase.
target_ahrefs_raw.columns = [col.replace(' ','_') for col in target_ahrefs_raw.columns]
target_ahrefs_raw.columns = [col.replace('.','_') for col in target_ahrefs_raw.columns]
target_ahrefs_raw.columns = [col.replace('__','_') for col in target_ahrefs_raw.columns]
target_ahrefs_raw.columns = [col.replace('(','') for col in target_ahrefs_raw.columns]
target_ahrefs_raw.columns = [col.replace(')','') for col in target_ahrefs_raw.columns]
target_ahrefs_raw.columns = [col.replace('%','') for col in target_ahrefs_raw.columns]
While not strictly necessary, I like to use the count column as a criterion for aggregation, and if I need to group the entire table, I like to use the single-valued column “item”.
target_ahrefs_raw['rd_count'] = 1 target_ahrefs_raw['project'] = target_name target_ahrefs_raw
-
Screenshot from Pandas, March 2022
Now we have a dataframe with clean column names.
The next step is to clean up the actual table values and make them more useful for analysis.
Duplicate the previous data frame and give it a name.
target_ahrefs_clean_dtypes = target_ahrefs_raw
Clean up the dofollow_ref_domains column, which tells us how many ref domains a sitelink has.
In this case, we’ll convert the dash to zero and then convert the entire column to an integer.
# referring_domains target_ahrefs_clean_dtypes['dofollow_ref_domains'] = np.where(target_ahrefs_clean_dtypes['dofollow_ref_domains'] == '-', 0, target_ahrefs_clean_dtypes['dofollow_ref_domains']) target_ahrefs_clean_dtypes['dofollow_ref_domains'] = target_ahrefs_clean_dtypes['dofollow_ref_domains'].astype(int) # linked_domains target_ahrefs_clean_dtypes['dofollow_linked_domains'] = np.where(target_ahrefs_clean_dtypes['dofollow_linked_domains'] == '-', 0, target_ahrefs_clean_dtypes['dofollow_linked_domains']) target_ahrefs_clean_dtypes['dofollow_linked_domains'] = target_ahrefs_clean_dtypes['dofollow_linked_domains'].astype(int)
First_seen tells us the date the link was first found.
We’ll convert the string to a date format that Python can handle, and then use it later to deduce the age of the link.
# first_seen target_ahrefs_clean_dtypes['first_seen'] = pd.to_datetime(target_ahrefs_clean_dtypes['first_seen'], format="%d/%m/%Y %H:%M")
Converting first_seen to a date also means that we can perform time aggregations by month and year.
This is useful as it is not always possible to get a link to a site every day, although it would be great for my own site if it did!
target_ahrefs_clean_dtypes['month_year'] = target_ahrefs_clean_dtypes['first_seen'].dt.to_period('M')
Link age is calculated by subtracting the first_seen date from today’s date.
Then convert it to number format and divide by a huge number to get the number of days.
# link age target_ahrefs_clean_dtypes['link_age'] = datetime.datetime.now() - target_ahrefs_clean_dtypes['first_seen'] target_ahrefs_clean_dtypes['link_age'] = target_ahrefs_clean_dtypes['link_age'] target_ahrefs_clean_dtypes['link_age'] = target_ahrefs_clean_dtypes['link_age'].astype(int) target_ahrefs_clean_dtypes['link_age'] = (target_ahrefs_clean_dtypes['link_age']/(3600 * 24 * 1000000000)).round(0) target_ahrefs_clean_dtypes
Screenshot from Pandas, March 2022After cleaning up the data types and creating some new data functions, the fun begins!
link quality
Evaluation of the first part of our analysis link qualitywhich aggregates the entire data frame using the describe function to obtain descriptive statistics for all columns.
target_ahrefs_analysis = target_ahrefs_clean_dtypes target_ahrefs_analysis.describe()
Screenshot from Pandas, March 2022So from the table above, we can see the mean, the number of referencing domains (107), and the variation (25th percentile, etc.).
The average domain rating (equivalent to Moz’s domain authority) of referring domains is 27.
Is that a good thing?
It’s hard to know without competitor data to compare in this market segment. This is where your experience as an SEO practitioner comes in.
However, I’m sure we can all agree that it could be higher.
How high to shift gears is another question.
Screenshot from Pandas, March 2022The table above can be a bit boring and difficult to visualize, so we’ll draw a histogram to visualize the authority of the referring domain.
dr_dist_plt = ( ggplot(target_ahrefs_analysis, aes(x = 'dr')) + geom_histogram(alpha = 0.6, fill="blue", bins = 100) + scale_y_continuous() + theme(legend_position = 'right')) dr_dist_plt
smallScreenshot by author, March 2022The distribution is heavily skewed, indicating that most referring domains have an authority rating of zero.
Above zero, the distribution appears fairly even, with the same number of domains for different privilege levels.
link age is another important factor in SEO.
Let’s look at the distribution below.
linkage_dist_plt = ( ggplot(target_ahrefs_analysis, aes(x = 'link_age')) + geom_histogram(alpha = 0.6, fill="blue", bins = 100) + scale_y_continuous() + theme(legend_position = 'right')) linkage_dist_plt
Screenshot by author, March 2022The distribution looks more normal, even though it’s still skewed, most of the links are new.
The most common link age appears to be around 200 days, which is less than a year, indicating that most links were acquired recently.
For the sake of interest, let’s see how this relates to domain authority.
dr_linkage_plt = ( ggplot(target_ahrefs_analysis, aes(x = 'dr', y = 'link_age')) + geom_point(alpha = 0.4, colour="blue", size = 2) + geom_smooth(method = 'lm', se = False, colour="red", size = 3, alpha = 0.4) ) print(target_ahrefs_analysis['dr'].corr(target_ahrefs_analysis['link_age'])) dr_linkage_plt 0.1941101232345909
Screenshot by author, March 2022The plot (along with the 0.19 number printed above) shows no correlation between the two.
Why should there be?
Relevance simply means that a higher authority link was acquired earlier in the site’s history.
Irrelevant reasons will become more apparent later.
We will now look at link quality over time.
If we were to plot the number of links verbatim by date, the time series would look rather messy and not very useful, as shown below (code for rendering the graph is not provided).
To do this, we’ll calculate a running average of domain ratings by month of the year.
Note the expand( ) function, which instructs Pandas to include all previous rows in each new row.
target_rd_cummean_df = target_ahrefs_analysis target_rd_mean_df = target_rd_cummean_df.groupby(['month_year'])['dr'].sum().reset_index() target_rd_mean_df['dr_runavg'] = target_rd_mean_df['dr'].expanding().mean() target_rd_mean_df
Screenshot from Pandas, March 2022We now have a table that we can use to provide charts and visualize them.
dr_cummean_smooth_plt = ( ggplot(target_rd_mean_df, aes(x = 'month_year', y = 'dr_runavg', group = 1)) + geom_line(alpha = 0.6, colour="blue", size = 2) + scale_y_continuous() + scale_x_date() + theme(legend_position = 'right', axis_text_x=element_text(rotation=90, hjust=1) ))
dr_cummean_smooth_plt
Screenshot by author, March 2022This is interesting because the site seems to be attracting high-authority links from the start (probably the PR campaign that kicked off the business).
Then it disappeared for four years and then reappeared again with a new link for a high authority link.
Link volume
Writing that title sounds good!
Who doesn’t want tons of (good) links to their site?
Quality is one thing; volume is another, which we’ll analyze next.
Much like the previous operation, we will use an extension function to calculate the cumulative sum of the links obtained so far.
target_count_cumsum_df = target_ahrefs_analysis target_count_cumsum_df = target_count_cumsum_df.groupby(['month_year'])['rd_count'].sum().reset_index() target_count_cumsum_df['count_runsum'] = target_count_cumsum_df['rd_count'].expanding().sum() target_count_cumsum_df
Screenshot from Pandas, March 2022Here’s the data, now the graph.
target_count_cumsum_plt = ( ggplot(target_count_cumsum_df, aes(x = 'month_year', y = 'count_runsum', group = 1)) + geom_line(alpha = 0.6, colour="blue", size = 2) + scale_y_continuous() + scale_x_date() + theme(legend_position = 'right', axis_text_x=element_text(rotation=90, hjust=1) )) target_count_cumsum_plt
Screenshot by author, March 2022We saw that link acquisition slowed down in early 2017, but steadily increased over the next four years before accelerating again around March 2021.
Again, it’s better to correlate it with performance.
Go one step further
Of course, the above is just the tip of the iceberg, as it is a simple exploration of a site. It’s hard to deduce anything that will help you rank in the competitive search space.
Below are some areas for further data exploration and analysis.
- Add social media sharing data to both target URLs.
- Correlate overall site visibility to running average DR over time.
- Plot the distribution of DR over time.
- Add search volume data on hostname See how many branded searches the referring domain receives as a measure of true authority.
- Join crawl data to the target URL to test for content relevance.
- link speed – The rate at which new links are acquired from new sites.
- Combining all of the above ideas Enter your analytics to compare with your competitors.
I’m sure there are not many ideas listed above, feel free to share them below.
More resources:
Featured Image: metamorworks/Shutterstock
