Selenium testing using checksums of html

Question

I am new at selenium testing and am writing a bunch of tests for a webpage that relies heavily on javascript user interaction.

At first I wrote a lot of assertions of the style

 If I press button A then
      assert number of visible rows = x,
      assert checkboxes checked are such
      assert title = bar
      .... [20 more]

and so on.

Then I switched to checksumming the HTML using MD5:

 If I press button A then 
     assert md5(html) = 8548bccac94e35d9836f1fec0da8115c.  

And it made my life a whole lot easier...

But is this a bad practice in any way?

example

• I know a HTML file a.htm is working correctly, I copy it as a_test.htm as a testcase I make all checksums using selenium in dictionary.txt ('show_all' : ' 8548bccac94e35d9836f1fec0da8115c, 'hide_all' :3fdec30c2731d22e2516b1cd1261a1e1, 'filter_by_id_click' : 3fdec30c2731d22e2516b1cd1261a1e1) and so on..
• The use cases are done in selenium (driver.findbuttonShowAll.click(), assert(md5(html)==dict['show_all])
• Further development that doesn't brake expected html output is safe, when assertion fails I diff the htmls...

UPDATE : Note that taking this approach can check dynamic behaviour, because of the fact that the md5 strings are build browserspecific from html in memory. The html is extracted from a selenium webelement that in its turn is taken from the current webdriver state.

I am not changing the way selenium handles things, just comparing the underlaying html.

To get dynamic content, I pass the loaded webdriver to a helper method that extracts the outerHTML of a given xpath expression (python)

def get_outerHTML_by_xpath(p_driver, p_xpath_expression):
    webelement = p_driver.find_element_by_xpath(p_xpath_expression)    
    outerHTML= p_driver.execute_script("return arguments[0].outerHTML" , webelement)    
    return outerHTML

Answer 1

Peter, welcome to SQA. Rather than label the MD5 approach bad or not bad, it may help to think about how it relates to an image capture approach, i.e. an approach in which you compare browser screenshots. I do not mean to imply that these two approaches are the only ones worth considering; rather, they are worth comparing because both ignore the meaning of the page in exchange for a easily described, coarse-grained comparison.

• External resources. The MD5 approach, as you stated it, ignores external resources such as stylesheets, images, Javascript, and applets, whereas the image approach captures the visual gestalt of everything loaded in the browser. Of course, with some extra work the MD5 approach could checksum those external resources, too.
• Hidden fields. The MD5 approach includes hidden fields in the checksum; the image capture approach does not. Hidden fields may be irrelevant to whether the page is visually correct, but knowing their values may help diagnose problems with how the page (or subsequent pages) behave. If a hidden field is dynamically generated, the MD5 approach may fail.
• Dynamic IDs. Some frameworks use dynamic IDs in the HTML. The MD5 approach fails with dynamic IDs; the image capture approach is oblivious to dynamic IDs.
• Other dynamic behavior. Some user interfaces incorporate the current time into the HTML, either visually or in hidden fields. Both MD5 and image capture approaches would fail with that kind of page.
• Browser dependencies. The same HTML (and external resources) may behave correctly in one browser but not in another. Of course behavior can vary among different versions of the same browser, too. The MD5 approach would tell you whether the right HTML is loaded into the page. Different browsers will not necessarily load resources in the same order (or load them at all), so your MD5 approach would need to deal with that. The image capture approach would require capturing images per browser, which would increase your maintenance cost.

Of course there are other ways to approach visual capture -- see Sikuli for example -- but that is beside the point, which is this: there is more than one way to approach coarse-grained comparisons, and each has their own trade-offs.

Another way to think about your question is in terms of how it would fit into your testing process. Somewhere in the process, someone tests the user interface manually to look for things that are hard to detect automatically, e.g. visual/layout problems, timing issues, and spelling/wording problems. Would that be the same person who maintains the automated tests? If not, how do you avoid duplicating work? Of course this problem arises with assertion-style automation too, but assertion-style testing does not attempt to account for visual issues, whereas you could argue that part of deciding that it is time to checksum a page is that the page is visually correct.