Hello,
Can you send me some of the website url which you would like to crawl, so that I can run some test. Also do let me know what fields (specifically) you are interested to collect from those urls.
How many such urls are you planning to crawl, coz if that number is too high, its wise to plan the solution scalable.
Regards,
Samiran