Building sample web crawling on AWS using Python

A proposal has not yet been provided

Dear Sir, I have reviewed your job requirement carefully and then excited. I have rich experience in scraping application for AWS. I have just delivered such a job to client from US recently, so I have already app to do it. It is written as C# not python. I recommend this app because speed is very fast than others. Let's discuss further detail. Sincerely, An

I read your requirements and I was happy to see that this is exactly my area of expertise! You did a good choice by choosing the scrapy framework. It is very stable, easy to learn, and fast! There is one alternative, called selenium framework, which allows to control a normal webbrowser from python, so it is helpful to scrape sites with high security measures. But on the sites you mentioned it shouldn't be needed. The timeline you've chosen seems very appropriate for this project to go smoothly. I say I deliver in 5 days, but thats just steps until step 3. After that you can take as much time as you need. I will give you support with any question relating to this project for as long as it takes. I'm eager to start! Hope you choose me, you won't be disappointed.

A proposal has not yet been provided

I graduated from Carnegie Mellon University with a master degree. I have lots of industry experience in big data area. I worked at IBM, Twitter before. CMU is the top 1 University in Computer science!

Dear Client: I can do the jobs using open-source Python/Scrapy framework. I have very python + web data scraping experiences in following tech/libraries/languages: • Parsing XML, HTML, JSON, JS code, text etc. • Hadoop/MR, nltk • Proxying, Delay/throttling, cookies • Scrapy • Python, lxml, XPath, beautifulsoup, urrllib, • mySQLdb, xlrd, xlwt, csv, minidom, Image, • Smarty, PHP, C/C++, Java • Ruby, mechanize, nokogiri, scraping • Regex, JS/Ajax/JSON, html/xml, PyV8 • Csv, excel, mySQL • Selenium Webdriver/FF/Chrome, Xvbf, etc. • Linux/CentOS/Ubuntu, Windows I have scraped over 30s of websites containing XML/JS/Ajax/Dynamic data contents – some websites with multiple regions, countries, currencies. I have installed and configured Scrapy on several platforms: CentOS, Ubuntu, Windows. I am currently maintaining a Scrapy based web data capturing/harvesting platform on Ubuntu 12.x for a private US client. It is used to source products attributes and images, classify products, and determine prices of over 30,000 products of different categories (toys, books, medical devices, footwear, apparels etc.) from 15s of different websites (in multiple formats/feeds: HTML/XML/JSON, csv, Excel, PDF etc.) for feeding to an e-commerce site. The scrapers store the data directly in a mySQL database comprising 5 tables. Thanks, Malik.