Competitor E-commerce Data Scraping

I want a robust web-scraping solution that captures the full product catalogue from a handful of my direct competitors’ online stores (I’ll share the URLs privately; they are not Amazon, eBay or standard Shopify marketplaces). The goal is to pull every detail we need for pricing intelligence and content optimisation: current price, full description, category and sub-category path, all image URLs, plus every variant attribute such as colour, size or other options the site shows. The scraper should run on demand and be easy to schedule for weekly updates. Anti-bot countermeasures—rotating proxies, polite timing, and basic CAPTCHA handling—will be essential because some of these sites throttle traffic quickly. I’m comfortable with a Python stack, so tools such as Scrapy, BeautifulSoup, Selenium or Playwright are welcome as long as the final code is clean, well-commented and handed over in a private Git repository. Deliverables • • CSV or Excel export containing: product URL, title, description, category, sub-category, price, variant attributes, image links • Compressed folder (or cloud bucket) of downloaded images organised by SKU • A short README explaining how to rerun the scraper, adjust target URLs, and set the update schedule Acceptance criteria: the script completes a full crawl of one sample competitor site without manual intervention and the data fields match what the site displays. If you have experience bypassing rate limits and keeping scraped data well-structured, let’s get started.

Project ID: 40403424