2 Ways to Smash Scrapers and Hacks with Wordfence – wikiHow

Wordfence is a popular WordPress security plugin. Among the features are scanners that monitor hacked files and a firewall with regularly updated rules that proactively block malicious bots.

There’s also a useful feature hidden in the tool that makes available user-configurable firewall rules that can enhance your ability to block hackers, crawlers, and spammers.

For some reason, the tool isn’t immediately visible and you have to click through multiple menus to find it.

But once you find it, you’ll find an easy and effective way to stop crawlers, hackers, and spammers from attacking your website.

Scrapers are especially troublesome because they copy your content and publish it elsewhere.

Now, with the tools provided by Wordfence, you can do a few things with these scrapers.

Using a tool like Wordfence can help reduce the amount of content that crawlers plagiarize.

There are many WordPress security plugins and SaaS solutions to choose from, including Sucuri Security and Cloudflare, highly recommended. Wordfence is one of the many security solutions available, and it’s up to you to decide which one feels more comfortable in your workflow.

Wordfence and other solutions work well as a set it and forget it solution.

However, in my experience, I’ve found that the user-configurable firewall in Wordfence provides an opportunity to enhance the bot’s attack capabilities and really stick it to hackers and crawlers.

But before you dial the firewall, it’s important to understand how far these firewall rules can go, and we’ll look into that as well.

Wordfence WordPress Security

Wordfence is trusted by over 4 million users to protect their WordPress sites.

The default firewall behavior is to prevent bots from crawling too many pages too fast, or the activity displayed by bots and humans indicating an intent to hack a website.

The firewall blocks the malicious bot’s IP address for a period of time, after which Wordfence drops the blocking.

The default settings on the firewall work fine.

But sometimes bots can still pass and be able to crawl a site or probe its vulnerabilities by slowly crawling it.

A common method used by hackers is to set up a bot to quickly attack the site, and when it gets blocked, it takes turns to other IP addresses and user agents, which causes the firewall to restart the detection process.

But these bots aren’t always programmed very well, which makes blocking them more effective than using the default Wordfence settings.

Background information on Wordfence firewall rules

Effective bot blocking can be accomplished using server-level tools, multiple plugins, or even using .htaccess files.

But editing the .htaccess file can be tricky because there are strict rules to follow, and mistakes in the .htaccess file can cause the entire site to fail.

Using firewall rules is just an easier way to block bots.

What can you block with Wordfence?

Wordfence allows you to create blocking rules for each of the following reasons:

• IP address range
• CPU name
• browser user agent
• recommender

IP address range

An IP address is the IP address of the server or ISP from which the robot or human comes from.

CPU name

Hostname represents the name of the host. The host is not always declared, and sometimes a bot/human visitor is only shown an IP address.

browser user agent

Each site visitor usually tells the server what browser it is using. The browser user agent refers to the browser a visitor says it is using. Bots can say it’s pretty much any browser, and they sometimes do this to evade detection.

recommender

This is a page from which a bot or human allegedly clicked the link.

Wordfence custom mode blocking

The way to block malicious bots using any of the above four variables is to add a custom rule in the custom pattern blocking tool.

Here’s how to get there.

step 1

Click the Firewall link in the admin menu on the left side of WordPress

Step 2

Select the label labeled Blocking

Step 3

Select the Custom Mode tab and create firewall rules in the appropriate fields. One of the fields is labeled “Block Reason”. Use this field to add a descriptive phrase, such as hostname, user-agent, or others. It will help you see all the rules you create by being able to sort by the type of block.

Step 4

Step 5

Make your rules by clicking the “Block visitors matching this pattern” button and you’re done.

Wordfence rules can use asterisks

as a wild card.

Should you block IP addresses with Wordfence?

Wordfence makes it easy for publishers to set firewall rules that effectively block bots.

It’s a blessing, but it can also be a curse. For example, using the Wordfence firewall to permanently block thousands of IP addresses is not efficient, and Wordfence may be used incorrectly.

It is OK to temporarily block IP addresses. Blocking an IP address permanently is probably not good, because as far as I can tell, from memory, it can bloat or slow down your WordPress installation.

Often, permanently blocking thousands or even millions of IP addresses is best accomplished using an .htaccess file.

Block hostnames with Wordfence

Using Wordfence to block hostnames can be a way to stop hackers, spammers, and crawlers. You can view Wordfence real-time traffic logs by clicking Wordfence > Tools.

This shows you both bots and human visitors, including bots that were automatically blocked by Wordfence.

Not all website visitors display their hostname. However, in some cases they do display their hostname, which makes it easy to block an entire web host.

For example, a site, for whatever reason, attracts DDOS-level bot traffic from a single host. None of my other sites have gotten much attention from this host, just this one.

Between March 2020 and December 2021, a site received over 250,000 attacks, each of which was blocked by Wordfence.

Obviously, blocking bots by hostname can be useful if you want to block cloud hosts that only send hackers and crawlers.

However, some hosts, such as Amazon Web Services (AWS), send both bad bots and good bots. Blocking AWS servers may also unintentionally block good bots.

So it’s important to monitor your traffic and be absolutely sure that blocking hostnames won’t backfire.

On the other hand, if you don’t need traffic from Russia or China, it’s easy to block hackers, crawlers, and spammers from both countries by creating firewall rules with the hostname field.

All you have to do is create a rule that blocks all hostnames ending in .ru and .cn. This will block all Russian and Chinese hostnames ending in .ru and .cn.

This is what you enter in the hostname field:
*.ru

*.cn

This does not mean that anyone is encouraged to use Wordfence to block Russian and Chinese bots by hostname. This is just an example to show how it’s done.

Block hackers and crawlers by user agent

Many rogue bots use old and outdated browser user agents.

After the Russian invasion of Ukraine, I noticed an increase in hacking bots using the Chrome 90 User Agent (UA) of the same set of web hosts. Usually bot traffic is different on different websites. So it stands out when they look the same on all my sites.

Whenever Wordfence automatically blocks these bots from visiting my site too quickly, the bots switch IP addresses and start visiting the sites over and over again.

So I decided to block these bots through their browser user agent (often abbreviated as UA). First I checked StatCounter website

to determine how many users worldwide are using Chrome 90. According to statistics from StatCounter, as of January 2022, the Chrome 90 browser has a market share of 0.09% in the United States.

As of this writing, the Chrome browser is at version 100. Considering that Chrome automatically updates browser versions for the vast majority of users, it’s hardly surprising that Chrome 90 is used, so using a Chrome 90 browser user-agent won’t prevent really legitimate people from visiting your site.

So I’m sure it’s safe to use the Chrome 90 user agent to block anything that appears on my site.

However, there are some online tools, such as GTMetrix and Security Server Header Checker, which use the Chrome 90 user agent.

So if I block all versions of Chrome 90 (by using this rule: *Chrome/90.*), I also block both online tools.

Another way is to look at the specific Chrome 90 variants and online tools used by hackers.

Chrome/90.0.4430.212

GTMetrix and other tools use this Chrome UA:

Chrome/90.0.4400.8
Chrome/90.0.4427.0
Chrome/90.0.4430.72
Chrome/90.0.4430.85
Chrome/90.0.4430.86
Chrome/90.0.4430.93

Hackers and crawlers use these Chrome UAs: So if you want to allow online tools to still scan your site but also block bad bots, this is a example

*Chrome/90.0.4400.8*
*Chrome/90.0.4427.0*
*Chrome/90.0.4430.72*
*Chrome/90.0.4430.85*
*Chrome/90.0.4430.86*
*Chrome/90.0.4430.93*

How do i do this:

How to Block Chrome 90 with Wordfence

Warning about blocking user agents

Before blocking Chrome 90, I’ve been checking Wordfence traffic logs (accessible via Wordfence > Tools) to make sure no legitimate bots like GTMetrix are using Chrome 90 using that user-agent.

For example, you probably don’t want to block Chrome 96 because some of Google’s tools use Chrome 96 as the user agent.

Always research whether legitimate bots use a specific user-agent or hostname.

An easy way to do research with Wordfence traffic logs.

Wordfence traffic logs

Wordfence traffic logs show at a glance all user agents visiting your website in near real time. Traffic logs show information such as user agent, indicate whether a visitor is a bot or a human, provide IP addresses, hostnames, pages being visited, and other information that helps determine whether a visitor is legitimate.

The way to access traffic logs is to click Wordfence > Tools.

Blocking old browser versions is an easy way to stop many bad bots. Chrome versions of the 80, 70, 60, 50, 30, and 40 series are particularly heavy on some sites. This is a example

*Chrome/8*.*
*Chrome/7*.*
*Chrome/6*.*
*Chrome/5.0*
*Chrome/95.*
*Chrome/5*.*
*Chrome/3*.*
*Chrome/4*.*

How to block old Chrome UA used by bad bots:

Again, the above does not encourage blocking the aforementioned bots.

The reason I would use *Chrome/6*.* is because with one rule I can block the entire Chrome 60 series user-agents, Chrome 60, 61, 63, etc. without writing all ten user-agents.

I can block the entire 60 series with one rule. Don’t block the tenth and above series like this *chrome/1*.*

Because this also blocks the latest version of Chrome, Chrome 100. The above is a example

How to block malicious bots using the Chrome user agent described. Bad robots also use old and retired Firefox browser user agent, some even show python requests/

as user agent.

Be careful when creating firewall rules

Always do your research first to determine what bad bots are being used on your own website and make sure no legitimate bots or website visitors are using those old and retired browser user agents.