Scrapy Web Crawling Help
€12-18 EUR / Stunde
Issue explanation:
Everything work just fine, except a crawl order, I add a priority method but didn`t work correctly.
Need to first write all author data, then all album and songs data and store to DB with this order.
I want to query items in a MySQL table by order from item in another one and make a better performance. On other words - need to rewrite some part of item pipeline (process_item method) to this work correctly.
Example:
First write all author items in Author table, and then order album items in Album table by authorId from Author table.
Thanks write other tables (what works) and go to another pages.
P.S. I didn`t know what missed, maybe Spider code need to be refactorized for different order.
Always is issue with a select query from another table and comparison like on example below.
(SELECT AuthorId FROM author WHERE AuthorName ='" + str([login to view URL]("albumAuthor")) + "')
Also I need to remove a duplicated items from a some attributes. Have a three item class for author, album and song parser. I was tried to make a another flow of spider and put all in one item class, but with no success. Order was a same. :(
Database structure: [login to view URL]
Github repository: [login to view URL]
Projekt-ID: #20759838
Über das Projekt
8 Freelancer bieten im Durchschnitt €14/Stunde für diesen Job
Hello. I am very familiar with Web scrapping. As a web expert. I have strong skills and experiences in development for 7 years. I hope to work for you because my skill set is very suitable for your project. Will wait f Mehr
Hi, I can rewrite parts of your script to make your pipeline work perfectly. I have done many projects in python , web scraping ,mysql Some of my projects include 1. Instagram Automation This tool does complete a Mehr
Hi I read your job description carefully and I can do your job perfectly. I have scraped many websites using python. I used to use BeautifulSoup or Selenium for web scraping. So I can know what you mean and I am ready Mehr
Dear sir, I am able to deliver you the high-end quality work according to your requirements. I will deliver to you the different creative ideas that will meet your satisfaction level. I believe in original and creative Mehr
Hi there! I am 4+ years experienced developer as Python, Django, RoR & ReactJS. Please open the chat box for further discussion. Regards,
Hi, Thanks for sharing your requirements here. I’m an Experienced Freelancer with a demonstrated history of working in the internet industry. Skilled in Core PHP, MySQL, HTML, CSS, JQuery, JavaScript. I have worked m Mehr
