Not all bots are bad. Like the one bad apple, the malicious botnets used by cyber attackers to scan and steal confidential user or product data from websites have spoiled the reputation of good bots like search engine crawlers, SEO and online advertising bots. But there’s no denying, there are a lot of bad bots out there. Bad bots provide no benefit to your business or website (search engine ranking for example) and typically tie up valuable CPU resources and run endless executables at best – or perform content scraping at worst.
The knee-jerk reaction has been to block all bots from interacting with a website to scrape information, but that can do more harm than good if blocking prevents Google, SEO and online advertising bots from doing their legitimate jobs. If blocking isn’t the answer, what is? Bot management is emerging as a proactive best practice that gives businesses greater visibility and control over bot activity to lessen the impact on the business and website performance.
Both good and bad bots alike are a pain point for already strapped IT departments that are challenged to detect, identify, and analyze the wide range of bot activities, which span the beneficial to downright dangerous. As a result, most IT departments operate with a constant fear of bot activity, regardless if it is good or bad and adopt a defensive strategy akin to the “Whack-A-Mole” arcade game. Once detected, it is common to systematically deny the bot activity, unintentionally blocking many of the good bots.
Fortunately, the emergence of new bot management solutions as part of a web security infrastructure is providing businesses with a new option for protecting themselves against malicious botnets while still allowing search engines and other legitimate bots to retrieve information.
Why bot management matters – and why IT and eCommerce managers are starting to transition from a mindset of bot blocking to bot management – can be summed up in three key advantages:
Detect and identify a bot in real-time
Knowledge is always power when it comes to web security, and the ability to quickly detect and identify bot traffic puts IT at the head of the class. An effective bot management strategy starts with an understanding of the different types of bots and how each can impact the business for good or bad. Obviously, you want to fight back against malicious web content scraping, price scraping, inventory grabbers, personal information harvesters, and the like, that can damage your competitive advantage. But you also want to maximize the benefit from legitimate good bots.
Even legitimate bots that benefit your business, like those operated by partners or contracted third-party services, may behave just like their bad counterparts when it comes to targeting certain information or generating aggressive traffic – so you don’t want to block them accidentally. The right bot management solution will enable IT to identify the various good bots by category, define acceptable bot signatures, and analyze the type of activities performed by the bot. In turn, the solution’s visibility and reporting capabilities help IT analyze all bot traffic to more accurately determine if the intention is malicious or not.
Gain control over malicious bot activity
The next advantage is gaining control over malicious bot activity. Your typical bad bots are automated clients whose intent is to steal content and intellectual property, conduct fraud, and spam websites and their legitimate users. But don’t be fooled by the word “automated.” There is a human operator behind every web scraper, inventory grabber, and any other bad bot with the motivation to steal information that will give them a financial or competitive advantage. Every time IT blocks the bot, the operator will know and will update the bot signature to try again. Keep on blocking them and they will continue to return each time more stealthily than before, eventually making it difficult if not impossible to detect them. A bot management solution should provide the ability to control malicious bot traffic without alerting the operator, such as by slowing down the rate at which the bot can retrieve information or by feeding it different information than what it came for. You’re keeping the bad bot at bay while minimizing its impact on your web infrastructure and your business.
Reduce the impact of bots on website performance and the business
You want to give preferential treatment to legitimate bots like Google, partners and contracted third-party services, but bot management lets you stay in control of how their activities impact website performance. Even good bots can have bad habits, like eating up resources during key business hours or during holiday shopping time, for example. A robust bot management solution can help IT manage good bot traffic based on the impact to the business at any given time.
For example, an eCommerce site might provide its resellers with permission to use web scrapers to collect current product information. However, it might want to automatically slow down the reseller’s bot traffic during peak shopping times when online customers expect a consistent user experience with no site delays. As a result of redirecting or reducing bot traffic, eCommerce businesses can ensure peak website performance and minimize lost sales and customer churn. What’s more, the added security of a bot management solution to protect against malicious price scrapers and content aggregators can pay off in terms of protecting the business’s competitive advantage.
Conclusion
The bottom line is pretty simple: refrain from blocking bots as much as possible and consider alternate behavior instead for a more efficient bot management strategy. Manage them with a proactive bot management solution that enables eCommerce sites and other online organizations to gain greater visibility into the bots that access their sites and greater control over the actions they’re taking. This requires a set of capabilities working in concert to provide automation and intelligence that can be used by human analysts for the most effective result.
This approach works best as a strong complement to a comprehensive web application security strategy that includes a web application firewall (WAF). Most of all, the most effective bot management solution will give IT the widest and most flexible range of options to apply to controlling bot traffic so that the business – and all of its legitimate customers and business partners – will be minimally impacted by malicious bot activity.
This article was produced in partnership with Akamai. For more information, please go here.
