Body Text Extraction

Body Text Extraction Articles published on the WWW often contain extraneous clutter. Most articles consist of a main body which constitutes the relevant part of the particular page. Surrounding this body is irrelevant information such as copyright notices, advertising, links to sponsors, etc. The technology needed to extract only relevant text is commonly used in clustering and categorization applications where html is used. Below are a couple of URLs to one example we have found for solving the body text extraction problem, but the tool is written in Python and Perl. Do not limit your bids to just this solution if you have other ideas or experience to offer. <[login to view URL]> <[login to view URL]> 1. We need a Java function 2. That can be embedded within our tool 3. That can take an HTML based document that is currently stored in memory and produce just the main body of text 4. No text associated with buttons, advertisements, etc. 5. The function needs to work on all types of HTML based documents We are NOT looking for a HTML stripper or HTML to text converter. We are less concerned about the format and the chance of loosing some information, than we are that it contains the majority of the main context of the document. Our tools will then learn from the context and knowledge within the extracted text. Please provide a bid along with a high level description on how you will solve this problem. Also provide information on any experience you may have in this area of work. ## Deliverables 1) Complete and fully-functional working program(s) in executable form as well as complete source code of all work done. 2) Deliverables must be in ready-to-run condition, as follows (depending on the nature of the deliverables): a) For web sites or other server-side deliverables intended to only ever exist in one place in the Buyer's environment--Deliverables must be installed by the Seller in ready-to-run condition in the Buyer's environment. b) For all others including desktop software or software the buyer intends to distribute: A software installation package that will install the software in ready-to-run condition on the platform(s) specified in this bid request. 3) All deliverables will be considered "work made for hire" under U.S. Copyright law. Buyer will receive exclusive and complete copyrights to all work purchased. (No GPL, GNU, 3rd party components, etc. unless all copyright ramifications are explained AND AGREED TO by the buyer on the site per the coder's Seller Legal Agreement). ## Platform All that run Java