i need an advanced scraper that can extract all sub pages associated with a particular domain.
only those pages followed by search spiders need to be extracted.
output should be via .csv
multi-threading would be helpful.
should be able to crawl 1000 sites at a time.
desktop application is preferred. (not in java)