AlexT
01-26-2006, 01:01 PM
Hi Darren,
iSiloX/XC already supports settings for maximum HTTP values, such as MaxConnectTime, MaxSendTime, etc. What would be cool to have is minimum HTTP values as well. What I mean by that is for example to tell iSilo to wait at least x seconds before parsing the next link. A lot of sites catch iSilo being a "crawler" because it's too fast hitting their servers.
Best,
Alex
iSiloX/XC already supports settings for maximum HTTP values, such as MaxConnectTime, MaxSendTime, etc. What would be cool to have is minimum HTTP values as well. What I mean by that is for example to tell iSilo to wait at least x seconds before parsing the next link. A lot of sites catch iSilo being a "crawler" because it's too fast hitting their servers.
Best,
Alex