I am adding cloudflare to my website - this is a service that will hopefully root out unwanted bots and increase security. It will take time and I'm not sure that I will be able to get the website up and running this month but things are happening behind the scenes.

@D1001 - I appreciate your frustration!

Hopefully CloudFlare can filter-out those FaecesBorg bots (and others).

I appreciate this might be relevant (if you are already commited to a hosting company) - but for anyone hosting their own webservers, or in charge of their own network, a very good and cheap solution is to use PiHole for all DNS lookups, as that acts as a sinkhole for a lot of bot traffic.

On 15/03/2024 at 12:47, KeithMacdonald said:

@D1001 - I appreciate your frustration!

 

Hopefully CloudFlare can filter-out those FaecesBorg bots (and others).

 

I appreciate this might be relevant (if you are already commited to a hosting company) - but for anyone hosting their own webservers, or in charge of their own network, a very good and cheap solution is to use PiHole for all DNS lookups, as that acts as a sinkhole for a lot of bot traffic.

Thank you Keith. I will look in to that if/when I move forward with BRDatabase.

Webpage scraping is unfortunately rife, the main culprits being the large tech firms filling their boots to bolster their AI credentials.

Many web sites are now adding nofollow and noindex tags to prevent such theft.

https://www.lumar.io/blog/best-practice/noindex-disallow-nofollow/

(But I suspect you probably know about this anyway.)