Scrapy Jobs

I need a reliable, one-time scrape of roughly 300 public education websites that all share a similar page structure. Each site lists between 5 and 200 staff contacts (average ≈50). For every person you find, capture four fields: • full name • job title • email address • exact source URL where the data appears All pages are publicly accessible—no authentication hurdles—so the script can run headless without session handling. Please deliver: 1. A consolidated Excel file (.xlsx) containing every contact, with clear column headers and either a “Site” column or separate tabs—whichever keeps the data easiest to filter. 2. The scraper’s source code (Python with Scrapy, BeautifulSoup, or similar is fine) plus a brief REA...

I need a small, reliable program that can scan Google Search results, spot which ads are running, and export the findings to a CSV. The workflow is simple: I type a niche and a city, hit run, and the tool crawls Google for professional-service businesses in major cities only. For every advertiser it detects, the CSV must list business name, Country, city, street address, email, phone, primary contact, and an estimated PPC budget. On top of that, the scrape should extract key campaign insights: • Budget • Keywords used • Ad placements A straightforward command-line script in Python is fine—Selenium, Scrapy, BeautifulSoup, SerpAPI, or the Google Ads API can all be leveraged so long as the solution stays within Google’s terms and reliably handles captchas...

No worries, that's way more workable. Here's the full post trimmed to fit under 10,000 characters: Nationwide Property Auction Web Scraping & Intelligent Alert System (Ongoing) About Us We're a commercial real estate investment firm that acquires distressed properties nationwide. We have the capital to close on any deal in the U.S. — our bottleneck is finding opportunities before competitors. We're building an automated system that monitors every property auction source in the country, filters against our criteria, and alerts us only on qualified deals. This is not a data dump project. We don't want spreadsheets with thousands of rows. We want a smart radar system that scans everything, filters ruthlessly, and only pings us when something matches. Long-t...

I need a small, reliable program that can scan Google Search results, spot which ads are running, and export the findings to a CSV. The workflow is simple: I type a niche and a city, hit run, and the tool crawls Google for professional-service businesses in major cities only. For every advertiser it detects, the CSV must list business name, Country, city, street address, email, phone, primary contact, and an estimated PPC budget. On top of that, the scrape should extract key campaign insights: • Budget • Keywords used • Ad placements A straightforward command-line script in Python is fine—Selenium, Scrapy, BeautifulSoup, SerpAPI, or the Google Ads API can all be leveraged so long as the solution stays within Google’s terms and reliably handles captchas...

Florida Judiciary Web Scraper — Config-Driven, Resilient Architecture I need a Python-based web scraping application to collect judge data from all 20 Florida judicial circuits and output it to a standardized CSV. The tool must be built for long-term maintainability — when a circuit website changes layout, only minimal configuration updates should be needed, not code rewrites. Background: Florida has 20 circuits covering 67 counties. Each circuit publishes judge data differently: some offer Excel/CSV downloads, others publish HTML pages and subpages with varying structures. The master data source is: Required Output Fields: (CSV)ID, Type, Name, Lastname, Assistant, Phone, Location, Street, City, State, Zip, County, Circuit, District, Courtroom, Hearingroom, Subdivision(S...

I need a lightweight, repeatable scraper that gathers every publicly visible customer review talking about Bayer from social-media sources—right now the focus is on Goole. The crawler should pull the full review text, star rating (or reaction score, if available), reviewer name or handle, date, and the direct URL to each post. Please build it so I can run it on demand, ideally from a simple command line or Jupyter notebook. Python with requests / BeautifulSoup, Selenium, or Scrapy is fine; if you prefer another stack, let me know why it would be a better fit. Deliverables • Clean, well-commented source code • One sample export in CSV or JSON showing at least 100 live reviews • A short README explaining environment setup, run instructions, and how to alter s...

I need a Python-based scraper that pulls complete car-listing information from every day. At a minimum the script has to capture make, model, price, and mileage but, in practice, I want every publicly visible field on each listing so that nothing useful is missed. Here’s what matters to me: • Reliability – the code must navigate pagination, work around basic anti-bot measures (rotating user-agents / respectful delays), and throw clear errors if the site layout changes. • Clean output – save to CSV or an SQLite database with consistent column names, ready for later analysis. You’re free to choose libraries you trust (requests, BeautifulSoup, Selenium, Scrapy, Playwright, etc.); just document any setup steps and keep third-party dependencies to a mi...

No worries, that's way more workable. Here's the full post trimmed to fit under 10,000 characters: Nationwide Property Auction Web Scraping & Intelligent Alert System (Ongoing) About Us We're a commercial real estate investment firm that acquires distressed properties nationwide. We have the capital to close on any deal in the U.S. — our bottleneck is finding opportunities before competitors. We're building an automated system that monitors every property auction source in the country, filters against our criteria, and alerts us only on qualified deals. This is not a data dump project. We don't want spreadsheets with thousands of rows. We want a smart radar system that scans everything, filters ruthlessly, and only pings us when something matches. Long-t...

I need a Python-based solution that automatically gathers companies and shareholders data, pulls supplementary details via external APIs, and outputs a clean, unified dataset I can query at any time. Scope of the scrape • Sources: company websites, financial databases and relevant public records. • Website focus: company profiles, turnover figures and any available Demat / share-holding particulars. What the tool should do 1. Crawl or call the above sources, respecting and rate limits. 2. Parse the required fields, normalise names and IDs, then enrich each record through one or more APIs (for example OpenCorporates, Clearbit or any better suggestion you have). 3. Store results in a structured format (CSV plus an SQLite or Postgres option). 4. Offer a simple comma...

I need a reliable script or windows-application that automatically gathers text content from specified websites and online databases, then saves everything into a clean, well-structured CSV file. A Windows-software would be preferred. The crawler should be able to crawl the website and spider a list of urls for approval or automatically go through the website Or just scrape a given list of urls (from a txt-file) Key details • Sources: public-facing websites and shops (also with login using username:password) • Data type: text only—no images or binary files. • Output: one CSV per run, UTF-8 encoded, with a header row • should be able to read/exrtract data from !! various shops & websites !! -> generally i need a basic software + "plugins" fo...

We are looking for an experienced developer who can build an automated system to extract daily newly incorporated company data from the MCA (Ministry of Corporate Affairs) website – https://www.mca.gov.in. The system should automatically collect and deliver the list of companies incorporated each day in structured format (Excel / CSV / API / Database). Scope of Work: Develop a web scraping or API-based solution to extract daily incorporated company data from the MCA portal. The tool should automatically fetch newly incorporated companies every day. Data should include the following fields (minimum): CIN Company Name Date of Incorporation ROC (Registrar of Companies) State Company Type (Private Limited / LLP / OPC / Public Limited) Authorized Capital (if available) Regist...

We are looking for an experienced developer to build a robust web scraping solution capable of extracting structured data from a login-protected medical/drug repository website. The platform contains a large database of drug information (potentially hundreds of thousands to over a million pages). The scraper should be able to navigate through the website after login, systematically extract relevant drug data, and store it in a structured format. Scope of Work: Develop a scraper that can log into a protected website. Navigate through the drug repository pages. Extract structured information from each drug page. Handle pagination and large-scale crawling. Implement mechanisms to prevent crashes or interruptions during long scraping runs. Store extracted data in a structured format such as ...

I have a curated list of specific company websites and I need an automated solution that extracts complete contact information from each one. The goal is to turn every URL into a clean, ready-to-use lead. WEBSITE : The scraper should capture: • Email addresses • Phone numbers • Mailing addresses • LinkedIn profile link • Location (city / state / country) • First and last name • Occupation / job title • Company name • Company website A well-structured CSV or Excel file is the preferred output, with each field in its own column. I am comfortable with your choice of tech—Python with BeautifulSoup, Scrapy, or Selenium are all fine—as long as the script runs reliably and respects and rate limits where required. Ac...