Script / Tool for checking a large number of URLs -- 2

The goal of this project is to create a script, tool or program that is able to check if over 1 Million URLs are working or not.

The URLs are stored in a MySQL table with the following fields: url_id, URL, status, massage, and last_check. The field status stores the HTTP Status Code. A message can be an optional field to store an error message if the URLs is invalid or not exist. The last_check contains the timestamp of the last URL check.

The script/tool should use multi-threading to check URLs. The most important thing is to have a very fast tool to be able to check millions of URLs in a reasonable time. For example linkcheck 2.0.6 checks over 10,000 links within a few seconds. That should be our reference.

The script/tool should be able to run on a web server automatically (cron) and by a single trigger. In that case, url_ids which should be checked will be passed to the script/tool.

You are free to create this script/tool with python or any other programming language that will meet the requirements.


Readable and maintainable code

Fully tested (unit tests) with at least 80% code coverage

Use of Pylint (if you implement this with python)

Here are some reference implementations or similar tools:

[login to view URL]

[login to view URL]

[login to view URL]

Attached you'll find a CSV file with 1 Million URLs as an example for testing.

Fähigkeiten: MySQL, Python, Testen von Software, Web Scraping

Mehr darüber referenceerror: "adsmanagerapp" is not defined., link checker google ads, google link checker, adwords scripts, broken url adwords script, how to learn adwords scripts, url checker google ads, google sheets url checker, virtuemart large number items, virtuemart slow large number products, virtuemart large number products, reach large number people myspace, copy large number word excel, condition checking phone number greater using validations, layout grid large number columns, magento large number products, explain law large number, php script select country phone number, test script tool free, php script generate random digit number

Über den Arbeitgeber:
( 0 Bewertungen ) Esslingen am Neckar, Germany

Projekt-ID: #19236853

21 Freelancer bieten im Durchschnitt $480 für diesen Job


Hi, my name is Selim. I am from Solihull, UK. I read your `Script / Tool for checking a large number of URLs -- 2` project descriptions carefully before bidding. I checked your requirements as well... I dont see the a Mehr

$400 USD in 5 Tagen
(264 Bewertungen)

Hi, very nice to meet you ! I've great experience in Python. When you read my profile page, you can find that I'm a python expert. I've experience with similar project - it checks a link every 0.1seconds. Another Mehr

$1000 USD in 3 Tagen
(25 Bewertungen)

Hi, i have the job done with PHP (Multi Curl) check here please [login to view URL] input file [login to view URL] can you se Mehr

$450 USD in 1 Tag
(71 Bewertungen)

I have been working as a full stack web developer for over 5 years. I am highly skilled in Web and App so I feel confident that I can do your job without fail. I can start working right now and I will guarantee the q Mehr

$500 USD in 10 Tagen
(44 Bewertungen)

Hi there, I am java developer with 7+ years of experience. I have strong expertise in web crawling, web scraping, website monitoring, b-o-ts and web automation. I have already made lots of scraper and monitor tools Mehr

$600 USD in 5 Tagen
(38 Bewertungen)

Hello,sir. Thanks for your job. I am expert web scraping. I can do your job with BS4 and Seleinum framework of python. I can do any project in your demand completely by my good experiences of last ago. I think this Mehr

$300 USD in 4 Tagen
(32 Bewertungen)

Hello. How are you today? I've read your job post carefully. This job really attracts me and also your required skills are completely matched with my major skills. I have good knowledge and rich experience with it. I Mehr

$555 USD in 10 Tagen
(9 Bewertungen)

19 years custom data-mining /scraping multi-curl / php / mysql i think you forgot the db upload - but i would still like to take a look upload it when you get a chance thanks, joe

$750 USD in 10 Tagen
(67 Bewertungen)

Hi I need few things to work 1- I need to understand what is criteria of working URL - e.g. if it is opened without problems but contain blank screen - is it "working" url? If it redirects to some another url - is it Mehr

$340 USD in 4 Tagen
(10 Bewertungen)

Dear, sir @I am fulltime freelancer@ I read your description in carefully. I am python expert and I have enough experience for it i can implement using python for your job. If you hire me, I 'll provide good servi Mehr

$500 USD in 10 Tagen
(5 Bewertungen)

Hello. I can test this tool for you very quickly. I will be able to provide you exact tool in PHP or dotNet. Thanks

$500 USD in 7 Tagen
(2 Bewertungen)

Hello, how are you? I have checked the details. I have rich experience with MySQL, Python, Software Testing, Web Scraping. Please initiate the chat so we can discuss this job in detail.

$555 USD in 14 Tagen
(1 Bewertung)

Hello there, How are you doing? I have read the description, I have great experience doing similar jobs related to these skills MySQL, Python, Software Testing, Web Scraping. Please start the chat so we can have detail Mehr

$555 USD in 6 Tagen
(1 Bewertung)

Dear Client! I read your proposal carefully and I am really interested in your project. I am a senior developer and I can build your tool using any language such as c++, python, php, java and so on. I have rich ex Mehr

$500 USD in 10 Tagen
(1 Bewertung)

Hello i have worked on similar projects before. I am 100% sure i can done this project without problems using python 3.6 with requests module and bs4 to parse neccesary information required. Relevant Skills and Experi Mehr

$250 USD in 1 Tag
(0 Bewertungen)

i am an experienced software developer and i am suitable for this job because i have done this kind of work before as well and i won't disappoint you i will fulfill all of your specifications on time and in given bu Mehr

$277 USD in 2 Tagen
(0 Bewertungen)

I am pretty comfortable among CRON jobs and Asynchronous JS, which makes me confident about the task you proposed.

$555 USD in 10 Tagen
(0 Bewertungen)

Hi, Thanks for sharing your requirement here on freelancer and I will be more than happy to help you. Let me share with you my expertise. I have been working in MYSQL and Python for last 2 year and developed quite a Mehr

$355 USD in 10 Tagen
(0 Bewertungen)

Hi, As RPA developer we can make this happen we have already created a project like [login to view URL] can automate your process of checking urls and save the error and logs. please go through the video for my previous project. Mehr

$250 USD in 5 Tagen
(0 Bewertungen)

I have done this optimization work for a millions of html’s to check the content and I can replicate the same here.

$333 USD in 5 Tagen
(0 Bewertungen)