Web-scraping a big dataset

  • Status Geschlossen
  • Budget €30 - €250 EUR
  • Anzahl der Angebote 22


The task is simple. You have to go to this website [1] and download the publicly available data. Manually it would take forever. That’s why you will have to study the JavaScript that creates the requests. You are expected to write a script which makes POST requests and batch downloads the data.


Currently you can get 24H data for all stations. Using a script create requests for all days one by one (without overwhelming the server). Requests are processed and then the resulting data is dumped on an ftp server [2] with the respective request number. 24H data for all stations should be about 1.5 GB of data. Download them and put them on a server where we can access them in bulk.

Download procedure to be automated:

1) Enter an e-mail address

2) Select data format: Mini-Seed

3) Select time interval: Loop over A single day from 1/1/2009 to 16/10/2016

4)Select stations: Select all stations and all channels

5) Submit request

6) Wait and collect the completed request from [2]

[1] [url removed, login to view]

[2] ftp://[url removed, login to view]

Erhalten Sie kostenlose Angebote für ein Projekt wie dieses

Möchten Sie Geld verdienen?

  • Legen Sie Ihr Budget und Ihren Zeitraum fest
  • Stellen Sie Ihr Angebot kurz dar
  • Bekommen Sie Geld für Ihre Arbeit

Heuern Sie Freelancer an, die auch auf dieses Projekt geboten haben

    • Forbes
    • The New York Times
    • Time
    • Wall Street Journal
    • Times Online