techie

08 Jun 2010

We want the crawler of the search engine  to download as many resources as possible from a particular Web site. A crawler would  normaly ascend to every path in each URL that it intends to crawl. For example, when given a seed URL of  ****.org/a/b/page.     , it will attempt to crawl //b/, /a/, and /
Path-ascending crawler is effective in that they are very effective in finding isolated resources, or resources for which no inbound link would have been found in regular crawling.