Fail2ban jail to mitigate DoS attacks against Apache

Recently, one of our shared hosting webservers at Onlime GmbH got hit by a DoS attack. The attacker started a larger vulnerability scan against common Wordpress security issues. We already had common brute-force attack patterns on Wordpress covered by a custom Fail2Ban jail, which mainly trapped POST requests to xmlrpc.php or wp-login.php (the usual dumb WP brute-force attacks...). But this DoS attack had hundreds of customer sites as target and did not get trapped by our existing rules.

After having blocked the attacker's IP (glad this was no large-scale DDoS!), I wrote an extra Fail2Ban jail which traps such simple DoS attacks. It's a very basic Fail2Ban jail that should cover common attacks and should not cause any false positives as it is only getting triggered by a large amount of failed GET requests.

There are other good articles about setting up such Fail2Ban jails to block simple DoS, but they didn't quite fit our needs:

• Using fail2ban to mitigate simple DOS attacks against apache (or why I am a terrible sysop)
• Install fail2ban to protect your site from DOS attacks

Requirements

What we would like to accomplish:

• Scanning of all Apache access logs
• Ban the attackers IP if there are more than 300 GET requests during a time span of 5 mins resulting in HTTP non-200 (OK) status codes: 401, 403, 404, 503
• Ban the attacker for 1 hour

We're going to scan all customer Apache access.log logfiles which are in a slightly tuned (non-standard) combined log format:

#LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t %I %O \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %V %p" combined

Below failregex also works for default extended/combined or common Apache LogFormat.

Setup Fail2Ban jail

Let's call the new Fail2Ban jail apache-get-dos and create the following two files:

[apache-get-dos]
enabled  = true
port     = http,https
filter   = apache-get-dos
logpath  = /var/www/*/logs/access.log
datepattern = %%d/%%b/%%Y:%%H:%%M:%%S %%z
maxretry = 300
findtime = 5m
bantime  = 1h

In above jail, we didn't specify a banaction, as we're using the global default. You might want to put the following in jail.d/custom.conf:

jail.d/custom.conf

[DEFAULT]
bantime  = 300
findtime = 300
banaction = iptables-allports

# Fail2Ban filter to scan Apache access.log for DoS attacks

[INCLUDES]
before = common.conf

[Definition]
# Option:  failregex
# Notes.:  regex to match GET requests in the logfile resulting in one of the
#          following status codes: 401, 403, 404, 503.
#          The host must be matched by a group named "host". The tag "<HOST>" 
#          can be used for standard IP/hostname matching and is only an alias for
#          (?:::f{4,6}:)?(?P<host>[\w\-.^_]+)
# Values:  TEXT
failregex = ^<HOST> .*"GET (?!\/robots\.txt).*" (401|403|404|503)\s

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
#
ignoreregex =

You might want to tune the failregex to your needs. For our use case, I have created a regex101 with some sample log lines which you can play around with. It will ignore any failing GET /robots.txt requests (as most bots request this) and it will not care about "sane" HTTP status codes like 200 (OK), 301 (Moved Permanently), 302 (Found). So, please be aware, that this jail only protects you from some kind of DoS attack that targets a large amount of non-existing URLs/pathes, which is mostly the case for malicious vulnerability scans.

Monitoring jail

After having restarted Fail2Ban, you might want to monitor the new apache-get-dos jail:

# list status of apache-get-dos jail with monitored Apache logs
$ fail2ban-client status apache-get-dos
Status for the jail: apache-get-dos
|- Filter
|  |- Currently failed: 80
|  |- Total failed: 290517
|  `- File list:    /var/www/<user>/logs/access.log ...
`- Actions
   |- Currently banned: 0
   |- Total banned: 5
   `- Banned IP list:   

# review banning in fail2ban.log
$ grep -Pi 'apache-get-dos\] (un)?ban' /var/log/fail2ban.log
2021-09-14 17:45:47,285 fail2ban.actions        [107006]: NOTICE  [apache-get-dos] Ban 1.2.3.4
2021-09-14 18:45:45,503 fail2ban.actions        [107006]: NOTICE  [apache-get-dos] Unban 1.2.3.4

In case you have designed your failregex too aggressively and have banned too many IPs, remember those commands:

# unban an IP from a specific jail
$ fail2ban-client set <JAIL> unbanip <IP>

# unban an IP (or several) from all jails
$ fail2ban-client unban <IP> ... <IP>

# unbans all IP addresses (in all jails and database)
$ fail2ban-client unban --all

Bonus Commands

I'll provide you with some bonus commands to further investigate such DoS attacks:

# Top 20 list of IPs with a large amount of connections to the server
$ netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head -n20

You may also want to scan all Apache access logs for the IPs with most requests:

# Top 10 IP list for ALL sites for previous hour
$ grep -h "\[$(date -d -1hour +'%d/%b/%Y:%H:')" /var/www/*/logs/access.log | cut -d' ' -f1 | sort | uniq -c | sort -nr | head -n10

# Top 10 IP list for ALL sites for current hour
$ grep -h "\[$(date +'%d/%b/%Y:%H:')" /var/www/*/logs/access.log | cut -d' ' -f1 | sort | uniq -c | sort -nr | head -n10
 
# Top 10 IP list for ALL sites with filename
$ grep "\[$(date +'%d/%b/%Y:%H:')" /var/www/*/logs/access.log | cut -d' ' -f1 | sort | uniq -c | sort -nr | head -n10

I recommend you block such IPs directly on your front-end firewall. But in case you can't, use iptables to quickly block a single IP:

# Block a single IP
$ iptables -A INPUT -s <IP> -j DROP
# Unblock it
$ iptables -D INPUT -s <IP> -j DROP

And now, cross fingers you won't get hit by a brutal DDoS. I plan to cover DDoS attack mitigation in a future article.