What are web bots and are they bad?
Theres web bots crawling the internet, digging through all websites. Let's take a look at what they're and if you should care about them
Especially if you're doing web development or running a website, you'll most likely heard about bots, but might not be sure what they are
Bots are non-human visitors to a website, that might visit for a wide variation of reasons
Are bots bad?
Not necessary, but there certainly are a lot of bad bots on the internet and they can hurt you website in many ways
Ruining statistics
The first way, is to ruin your statistics. Especially if you do you own monitoring and don't add any filters for bots, it's impossible to see what is legitim and what's bot traffic in you statistic. Especially for new sites, majority of your visitors will be bots (good and bad)
If you use a 3rd party for statistics, such as Google Analytics, there most likely already exist some bot filters included by default (some might need to be turned on)
[snippet]Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)[/snippet]
Good bots normally declare themselves in User Agent as a bot and often include an URL, where they belong to
[snippet]Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36[/snippet]
While bad bots often try to imitated either human or legitimate bots, making them more difficult to spot. Our site detected above as a bot, by other measurements than user agent. As you can't tell that it is a bot, if you are only looking at the User Agent string
Scrape
Some bot also scrape your website, which means they'll take copy of all the data they can find. This can be used to make their own fake duplicate website, use your articles on their own site or look for vulnerability, they can use for a bigger attack
Fake advertisements clicks
Another aspect, which can heavily impact your site is fake advertisement clicks. In the short run, it will generate you extra revenue, but if you don't manage to stop the worst of it, it can lead to you being banned from showing advertisement. Google Adsense have pretty decent filters, so they'll be able to filter away some of the fake clicks, but it's still important that you try to prevent them as much as possible
Looking for vulnerabilities
Another aspect, is looking for vulnerabilities on your website. These can be all from trying know URL vulnerabilities, from common CMS systems to common URL developers might implement, such as /admin
These will often try site injects into login, comment and other forms. Looking for vulnerabilities your site might have
Put to cart
For those running eCommerce sites, many sites will reserve stock as soon as they're put in cart and before user actually paid for it. Bad bots can be adding to cart and holding inventory they never intend to buy. Causing legitimate buyers not being able to buy and can even cause the back end system to re-order more stock, even if items will never be bought
Ban all bots?
Considering so many bots are up to mischief, should one just ban all bots? For most website, you'll hurt yourself more by doing so and thats because you'll block out the good bots as well
Search engines
Main reason to allow bots, is so that good bots can crawl your website and index it on search engines such as Bing, Google, Yahoo and not limited to Yandex
Most of our traffic comes from search engine referral and for many sites this is crucial traffic
