Look for a program that systematically scans an entire html document.

Question

I am helping my friend with some medical records verification. In short, I have to through hundreds of medical records and pinpoint certain ones for verification. I want to be able to search through these hundreds of records. The problems is that the medical records can only be viewed in a protected and private HTML document, which means I cannot extract them to a pdf. I believe that the documents are in from a asp.net database.

My best hope is to literally have my computer take a picture of the entire webpage. The computer screenshots, scrolls down, screenshots, scrolls down, screenshots .... etc. Does anyone know any program that will automate this iterative process. Once I have a pdf, I can easily parse it into searchable text.

Any help is appreciated.

Answer 1

You can use Selenium Webdriver to automate this task. It can read the DOM of the HTML, and extract information from it; it controls a browser the same way you do, so anything you can do manually it can do (with a few exceptions). I would suggest looking into that rather than taking screenshots and parsing them as PDFs.

Answer 2

Selenium 2 (webdriver) is designed to do exactly what you need. It allows you to simulate whatever real users can do with a browser: log to a website, click on buttons, fetch content of HTML elements, and analyze the text using any tools necessary. You can use different kinds of locators to identify HTML elements to process (ID, name, CSS class, link text etc).

Advantage of having text from HTML elements over images is obvious: much easier to process programmatically.

Selenium has bindings for many languages, but IMNSHO Python would be the best bet, because it also has excellent text processing features. Including regular expressions, if you want superweapons, but you can mostly avoid it by smart parsing and/or splitting. Unless you are expert in another language, of course.