Web Scraping from ~20 websites: retrieve simple information from catalogs of biological data

• Status: Closed
• Preis: $300
• Einträge eingereicht: 3
• Gewinner: rkfcccccc

Winner

Kurzbeschreibung des Wettbewerbs

Dear all,

I am looking for a specialist in data scrapping/python who could help me retrieve information from a few websites, using a list of keywords or a uniform structure of web-pages of these websites.

In brief, there are ~20 websites that contain catalogs of germs and their optimal growth temperatures. I need to extract the name of these germs and the values of their optimal growth temperatures, then merge these data into one .csv file, remove duplicates, and make a few minor changes to this file to make it ready to use by others. This way, we can create a unified and the largest list of germs with experimentally determined growth conditions (instead of having 20 different and difficult-to-use websites). The rest are details.

How our work will be organized:

Step 1
I use this contest to choose a person who can do the work. To win this contest, you need to retrieve names of germs and their growth temperatures from these two databases and show me a random print screen (or any other evidence) of the output files:

Database A:
https://www.bcrc.firdi.org.tw/wwwbcrc/WDCM_RefStrain-tableSearch.jsp

You will see a table of germs (Trametes versicolor, Aspergillus brasiliensis, etc) in which each germ having a link to its unique web page. Here is the page that describes the first germ in the list, Trametes versicolor: https://catalog.bcrc.firdi.org.tw/BcrcContent?bid=36525

Each page of this database is organized uniformly, with germ’s names following the word “Organism:” and the optimal growth temperature value following the word “Growth conditions:”


The expected output file describing this database will be a simple .csv file with two columns:

Organism Temperature (°C)
Trametes versicolor 26
Aspergillus brasiliensis 24
Aspergillus flavus 24
and so on.



Database B:
https://www.sci.muni.cz/ccm/index.html

Here you can see a full catalog where you need to click a certain letter or number, pick a “microorganism” (=germ) to get to a page that contains the germ’s name (first two words on the page), and its optimal growth temperature (the last value on the page).

For instance, just go to the Alphabetical List of Species,
then pick the letter “A”,
and the pick the first germ in the list,
it will bring me to the page that contains the germ’s name “Granulicatella adiacens” and its growth temperature “37 °C”

As in the previous case, the output file for this database will be a simple .csv file with two columns:

Organism Temperature (°C)
Granulicatella adiacens 37
Abiotrophia defectiva 37
etc. etc. etc.


Step 2
When I see that you can deliver, we have a brief Zoom/Skype meeting to discuss the rest of the project. I award you the project and we work for a couple of days to finish everything up before I release the payment and we call it a deal.

Attached is the list of the remaining databases, so that you understand the scale and the complexity of the project. I am expecting to work together with you, brainstorming, suggesting shortcuts and solutions and accepting that some websites will be too hard to tackle.

Empfohlene Fähigkeiten

• Python
• Web Scraping