SEO Projects in Python

After starting to learn Python late last year, I’ve found myself putting into practice what I’ve been learning more and more for my daily tasks as an SEO professional.

This ranges from fairly simple tasks such as comparing how things such as word count or status codes have changed over time, to analysis pieces including internal linking and log file analysis.

In addition, Python has been really helpful:

• For working with large data sets.
• For files that would usually crash Excel and require complex analysis to extract any meaningful insights.

How Python Can Help With Technical SEO

Python empowers SEO professionals in a number of ways due to its ability to automate repetitive, low-level tasks that typically take a lot of time to complete.

This means we have more time (and energy) to spend on important strategic work and optimization efforts that cannot be automated.

It also enables us to work more efficiently with large amounts of data in order to make more data-driven decisions, which can in turn provide valuable returns on our work, and our clients’ work.

In fact, a study from McKinsey Global Institute found that data-driven organizations were 23 times more likely to acquire customers and six times as likely to retain those customers.

It’s also really helpful for backing up any ideas or strategies you have because you can quantify it with the data that you have and make decisions based on that, while also having more leverage power when trying to get things implemented.

Adding Python to Your SEO Workflow

The best way to add Python into your workflow is to:

• Think about what can be automated, especially when performing tedious tasks.
• Identify any gaps in the analysis work you are performing, or have completed.

I have found that another useful way to get started learning is to use the data you already have access to, and extract valuable insights from it using Python.

This is how I have learned most of the things I will be sharing in this article.

Learning Python isn’t necessary in order to become a good SEO pro, but if you’re interested in finding more about how it can help get ready to jump in.

What You Need to Get Started

In order to get the best results from this article there are a few things you will need:

• Some data from a website (e.g., a crawl of your website, Google Analytics, or Google Search Console data).
• An IDE (Integrated Development Environment) to run code on, for getting started I would recommend Google Colab or Jupyter Notebook.
• An open mind. This is perhaps the most important thing, don’t be afraid to break something or make mistakes, finding the cause of an issue and ways to fix it is a big part of what we do as SEO professionals, so applying this same mentality to learning Python is helpful to take any pressure off.

1. Trying Out Libraries

A great place to get started is to try out some of the many libraries which are available to use in Python.

There are a lot of libraries to explore, but three that I find most useful for SEO related tasks are Pandas, Requests, and Beautiful Soup.

Pandas

Pandas is a Python library used for working with table data, it allows for high-level data manipulation where the key data structure is a DataFrame.

The best way to get started with Pandas is to take a simple CSV of data, for example, a crawl of your website, and save this within Python as a DataFrame.

Once you have this store you’ll be able to perform a number of different analysis tasks, including aggregating, pivoting, and cleaning data.

import pandas as pd
df = pd.read_csv("/file_name/and_path")
df.head

Requests

The next library is called Requests, which is used to make HTTP requests in Python.

It uses different request methods such as GET and POST to make a request, with the results being stored in Python.

One example of this in action is a simple GET request of URL, this will print out the status code of a page, which can then be used to create a simple decision-making function.

import requests

#Print HTTP response from page 
response = requests.get('https://www.deepcrawl.com')
print(response)

#Create decision making function 
if response.status_code == 200:
    print('Success!')
elif response.status_code == 404:
    print('Not Found.')

You can also use different requests, such as headers, which displays useful information about the page such as the content type and a time limit on how long it took to cache the response.

#Print page header response
headers = response.headers
print(headers)

#Extract item from header response
response.headers['Content-Type']

There is also the ability to simulate a specific user agent, such as Googlebot, in order to extract the response this specific bot will see when crawling the page.

headers = {'User-Agent': 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'}
ua_response = requests.get('https://www.deepcrawl.com/', headers=headers)
print(ua_response)

#Beautiful Soup 
from bs4 import BeautifulSoup 
import requests 

#Request URL to extract elements from
url= 'https://www.deepcrawl.com/knowledge/technical-seo-library/'
req = requests.get(url)
soup = BeautifulSoup(req.text, "html.parser")

#Print title from webpage 
title = soup.title
print(title)

for link in soup.find_all('a'):
    print(link.get('href'))

segment_definitions = [
    [(r'\/blog\/'), 'Blog'],
    [(r'\/technical-seo-library\/'), 'Technical SEO Library'],
    [(r'\/hangout-library\/'), 'Hangout Library'],
    [(r'\/guides\/'), 'Guides'],
    ]

use_segment_definitions = True

def segment(url):
    
    if use_segment_definitions == True:
        for segment_definition in segment_definitions:
            if re.findall(segment_definition[0], url):
                return segment_definition[1]
        return 'Other'

df['segment'] = df['url'].apply(lambda x: get_segment(x))

def get_segment(url):
  
        slug = re.search(r'https?:\/\/.*?\//?([^\/]*)\/', url)
        if slug:
            return slug.group(1)
        else:
            return 'None'

# Add a segment column, and make into a category
df['segment'] = df['url'].apply(lambda x: get_segment(x))

df['category_match'] = df['old_category'] == (df['redirected_category'])
df['segment_match'] = df['old_segment'] == (df['redirected_segment'])
df['depth_match'] = df['old_count'] == (df['redirected_count'])
df['depth_difference'] = df['old_count'] - (df['redirected_count'])

internal_linking_pivot['followed_links_in_count'] = (internal_linking_pivot['followed_links_in_count']).apply('{:.1f}'.format)
internal_linking_pivot['links_in_count'] = (internal_linking_pivot2['links_in_count']).apply('{:.1f}'.format)
internal_linking_pivot['links_out_count'] = (internal_linking_pivot['links_out_count']).apply('{:.1f}'.format)
internal_linking_pivot