More than 40% of all internet traffic comes from bots, and a quarter of total internet traffic comes from malicious bots.
This is why it’s important to detect the presence of bad bots as soon as possible and manage their activities accordingly. In short, a functional bot mitigation strategy is crucial for any business and even individuals with an online presence.
What Is Bot Mitigation
A key aspect of bot mitigation is to identify the bot traffic and properly distinguish bots from legitimate users, but there are other aspects to bot mitigation we should consider.
First, it’s crucial to understand that not all bots are bad. Bots are by nature, just tools. They are computer programs that are programmed to execute automated processes without any human intervention. These bots can execute repetitive tasks at a much faster rate than any human user ever could, and so they aren’t necessarily good or bad, it all depends on how the process/task it performs.
With that being said, there are actually many good bots that are beneficial to our site, application, and/or business, like Google’s crawler bot. Yet, there are indeed bad bots operated by cybercriminals to perform many malicious tasks.
Thus, a crucial aspect of bot mitigation is distinguishing between good bots and bad bots based on signatures, behaviors, and other factors.
Another important aspect of bot mitigation is what we will do to malicious bot traffic once it has been properly identified.
Completely blocking the bot and denying it from accessing our site’s resources might seem like the best and most cost-effective approach at first glance, but it isn’t always the best approach in all situations.
Block or Not Block Bot Traffic
There are two main reasons why blocking bot traffic isn’t always the best approach.
The first has been briefly discussed above: we wouldn’t want to accidentally block good bots, and even worse, legitimate human traffic. This is an issue we know as false positives.
The thing is, today’s bad bots have become so sophisticated in masking their identities and impersonating human behaviors. Bot programmers are now really advanced and many have adopted the latest technologies, including AI to hide the bot’s presence.
So, even distinguishing between bot traffic and human users is already challenging enough, much worse differentiating between good bots and bad bots. When we aren’t sure about the identity of the suspected malicious bot, then blocking is not a good idea.
The second reason is that blocking will not stop persistent cyber criminals from attacking your site. They will simply modify the bot to bypass your current bot mitigation measures, and they may also use information you’ve accidentally provided, for example in your error messages when blocking the bot, in upgrading this malicious bot.
In such cases, blocking the bot can be counterproductive, and this is why there are other bot mitigation strategies you should consider.
Bot Mitigation Approaches To Consider
If blocking the bot isn’t always the best approach, what are the alternatives? Here are some bot mitigation techniques to consider:
1. Rate Limiting
A key principle to understand when mitigating bot activities is that bots run on resources, which can be expensive. Thus, all bot operators would like the bot to execute the tasks as fast as possible while also using as few resources as possible.
Rate limiting, or throttling, works based on this principle: by slowing down our reply to the bot’s requests (i.e. lowering bandwidth), we can significantly slow down this bot’s operation without letting it achieve its objective.
The hope is that by slowing it enough, the bot operator will be discouraged and will move on to another target.
2. Feeding Fake Data
Similar in principle to rate limiting, but here instead of slowing down our bandwidth, we’ll reply to the bot’s requests with fake content. For example, we can redirect the bot to a similar page with thinner or modified content to poison its data.
Again, by letting the bot wastes resources, the hope is that the attacker will simply move on to another website instead of persistently attacking yours.
3. Challenging The Bot With CAPTCHA
When we aren’t completely sure about the identity of a client (whether it’s a bot or a human user), a fairly effective approach is to challenge the client with CAPTCHAs or CAPTCHA alternatives.
Keep in mind, however, that CAPTCHAs are not a one-size-fits-all solution and might not be ideal in certain cases:
- The most sophisticated bots with AI technologies can effectively solve CAPTCHA challenges.
- While we can make the CAPTCHA more difficult and challenging for bots, it will also increase the difficulty for human users, which may ruin our site’s user experience
- With the presence of CAPTCHA farms, CAPTCHA isn’t effective in stopping persistent attackers who are ready to invest in the services of these CAPTCHA-solving farms.
It’s worth noting, however, while CAPTCHA isn’t bulletproof, it is still a fairly effective bot mitigation technique in various situations to defend against less sophisticated bots. Use it tactically and sparingly.
When To Block The Bot Traffic
Blocking the bot traffic altogether remains the most cost-effective approach in theory since we wouldn’t need to process the bot traffic and use our resources in any way.
However, blocking the bot traffic is only ideal if we have an adequately strong bot detection solution in place that can consistently distinguish between good bots and bad bots, and can keep detecting the presence of malicious bots even after they’ve been modified and improved.
An AI-based bot mitigation solution that is capable of predictive, real-time behavioral analysis is essential, and by investing in one, you’ll get the easiest and most effective bot mitigation solution to implement in protecting your business from various bot threats.