« previous next »
Pages: [1]   Go Down

Author Topic: Limit traffic to bots  (Read 1295 times)

Offline ZManGT

  • VIP Member
  • ***
  • Posts: 94
  • Karma: 12
Limit traffic to bots
« on: February 26, 2009, 07:59:53 AM »
I'm compiling stats on some of our websites (we are a hosting company) and I've noticed on some of our websites this month alone bot traffic is over 2.6gb per site. We have multiple (50+) sites running on each server cluster so that is a lot of wasted traffic for bots. Do any of you know of a way to limit the speed of bot traffic so that it doesn't cause actual users connections to be slow during periods of high usage? How do you all deal with bot traffic?
Logged

Offline jmelika

  • Administrator
  • Hero Member
  • *****
  • Posts: 341
  • Karma: 7
Re: Limit traffic to bots
« Reply #1 on: February 26, 2009, 09:49:26 AM »
How do you identify the bot?  I am guessing from your web servers' logs not Netscaler...  I'm thinking 2.6GB is not necessarily that high.  It comes down to around 64Kbps on average.  The main metric you should be looking at is how often does the crawler hit your servers.  If it's every once in a while with large amounts of data per page is different than more frequently with little amounts.  Does that make sense?

Just thinking out loud.
Logged

Offline ZManGT

  • VIP Member
  • ***
  • Posts: 94
  • Karma: 12
Re: Limit traffic to bots
« Reply #2 on: February 26, 2009, 10:21:26 AM »
I'm identifying the bots from the web logs. Although the 2.6 gb sounds low, that's for 1 site in a 1 month period. Multiply that by 700 sites and all of sudden we are wasting bandwidth and resources on bots.
Logged

Offline jmelika

  • Administrator
  • Hero Member
  • *****
  • Posts: 341
  • Karma: 7
Re: Limit traffic to bots
« Reply #3 on: February 26, 2009, 10:25:41 AM »
If you can identify the bots at the Netscaler level using a request header that it sends, you could possibly drop some of their requests.  Do they send any specific request header that is unique?  If so, then that's half the problem.  The second half would be how to drop "some" of the requests and not all.  Let me know if that's the way you want to go and I'll go pick at the rules and policies and see if I can figure something out for you.
Logged

Offline Paul B

  • Hero Member
  • *****
  • Posts: 193
  • Karma: 20
Re: Limit traffic to bots
« Reply #4 on: February 27, 2009, 01:23:33 AM »
Ver 9 code introduced a "rate limiting" function.... if you could identify the bot traffic in some way (even if it uses "public pages") then you might be able to do something.


Paul
Logged

Offline knabs

  • Contributor
  • *
  • Posts: 1
  • Karma: 1
Re: Limit traffic to bots
« Reply #5 on: June 27, 2009, 08:53:34 AM »
Here is a thought.

1] build a reponder policy to identify the bot and redirect to VIP only for BOTs

HTTP.REQ.HEADER("User-Agent").CONTAINS("adidxbot")

2] servers in the BOT vip could be tunned way down..

This is only thinking out loud as Version 9.x does indeed support rate limiting.

-t
Logged
Pages: [1]   Go Up
« previous next »