Spider Software Recommendations?

Briask · 15-09-2008, 10:29

Have to be very careful with crawling, quite a few of the crawlers out there hit sites for too much too often and very quickly you will find your crawlers ip and UA on various black lists

ddemetrius456 · 13-11-2009, 07:18

What a helpful bunch of good people .

pokerscript · 07-02-2010, 14:54

I would like to setup my own small search engine, a little like google but which is easy for me to customize, but i need a whole script which i can use to do this and to also spider sites.

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

15-09-2008, 10:29	#11 (permalink)
Briask Cool Newbie Join Date: Sep 2008 Posts: 34 iTrader: 0 / 0%	Have to be very careful with crawling, quite a few of the crawlers out there hit sites for too much too often and very quickly you will find your crawlers ip and UA on various black lists

13-11-2009, 07:18	#12 (permalink)
ddemetrius456 Newbie Join Date: Nov 2009 Posts: 16 iTrader: 0 / 0%	What a helpful bunch of good people .

07-02-2010, 14:54	#13 (permalink)
pokerscript Banned Join Date: Feb 2010 Posts: 16 iTrader: 0 / 0%	I would like to setup my own small search engine, a little like google but which is easy for me to customize, but i need a whole script which i can use to do this and to also spider sites.