|
About crawling forum threads
Link |
by kawaii_ryu
on 2006-01-06 18:31:27 (edited 2006-01-06 18:33:56)
|
|
Hi Gendou-sama I'd plan to crawl this website, its any rule about crawling with software ? i use teleport pro, its ok ? of course just the forum pages, not a download page, how to do this ? many thanks |
|
Re: About crawling forum threads
Link |
by
on 2006-01-06 18:34:58 (edited 2006-01-06 18:36:23)
|
|
any software that obeys robots.txt should be fine. however, if you plan on accessing more than 1,000 (which you probably will), please spread the crawl out over as long a period of time as possible, as a courtesy to other users. 
|
|
Re: About crawling forum threads
Link |
by kawaii_ryu
on 2006-01-06 18:45:18 (edited 2006-01-06 18:45:38)
|
|
So, i can't crawl all thread in same time to prevent server busy isn't ? ok, thanks btw, your rabbit gif is so cute, what a name that rabbit ? |
|
Re: About crawling forum threads
Link |
by
on 2006-01-06 18:56:20 |
|
i dont know the name of the rabbit, sorry.
|