We want the crawler of the search engine  to download as many resources as possible from a particular Web site. A crawler would  normaly ascend to every path in each URL that it intends to crawl. For example, when given a seed URL of  ****.org/a/b/page.