Abstract:

Based on the Web surf behaviors of human beings, crawling directions are restricted by extracted crawling subpages, and the associated relationships of crosspage compound object are  realized through the properties′ inheritance between crawl datum. Then, the generalized crawl process with deep directional collection support is  designed and implemented. Experimental results about the hot posts of the Tianya site show that this method can achieve data collection of complicated objects without changing the main procedure, and has high collection efficiency.

Key words: deep collection; directional web crawler; public web opinion