Data Mining / Cross-Referencing Script

I need a script that will harvest data from a website. Then the data need to be cross-referenced to data held on a different website. Step by step: 1) Log into website #1. 2) Navigate to a static HTML page. 3) Collect first name, last name, and city/village/township for each entry (usually about 30 entries or so). 4) Once that is done, navigate to website #2. 5) Search the new website for each entry using a search form on website #2. 6) The search results will give you addresses, these need to be collected. 7) Spit out a comma separated value text file of the data. Ok, so this is pretty simple, right? There are 2 problems. Problem #1 ... At step 3, the data isn't always in the same format. However, there seems to be a finite set of formats (looks to be less than 5 as best I can tell for now). So the data harvesting logic has to account for all the different formats. As well, some of the names include middile initials, while others don't, jr/sr suffix, etc. Problem #2 ... At step 6, the search results can be finicky. Sometimes you have to search partial names or with only parts of the data to get results. So the logic should probably search very specifically first, then step through broader searches until it gets results. I have a 90% working solution done for steps 1 through 3. And I've developed some logic for steps 4 through 7. However, the way I'm doing it just feels like brute force (I'm not a coder, I just play with scripts in my off time). I can show you website #1 and website #2 if you are interested in looking at them before responding. My code (written in Perl) is attached, I'm not expecting anyone to follow my format. In fact, I would expect something more elegant (if possible). I only spent a couple hours on it, and most of that was learning how to code in Perl, heh. The main while() loop (lines 25 through 103) is my funky searching algorithm (it's basically tip-toeing through the text to find what I want), in case it looks confusing. ## Deliverables Deliverables are easy. All I want is a Perl script that is well documented so I can understand what is going on in case I need to make modifications in the future. Or, if you have a suggestion for a better language to write this in, let me know. * * *This broadcast message was sent to all bidders on Thursday Oct 12, 2006 10:24:14 AM: I'm sorry to inform the bidders that this task has exposed some more complications in the text searching logic. While this would not present problems for the coding itself, it would present problems for developing the text searching rules. I am attempting to resolve these complications but cannot select a bidder at this time. ## Platform Perl script capable of running on Win XP (Perl is platform-independent, right?).