I am new at selenium testing and am writing a bunch of tests for a webpage that relies heavily on javascript user interaction.

At first I wrote a lot of assertions of the style

 If I press button A then
      assert number of visible rows = x,
      assert checkboxes checked are such
      assert title = bar
      .... [20 more]

and so on.

Then I switched to checksumming the HTML using MD5:

 If I press button A then 
     assert md5(html) = 8548bccac94e35d9836f1fec0da8115c.  

And it made my life a whole lot easier...

But is this a bad practice in any way?


  • I know a HTML file a.htm is working correctly, I copy it as a_test.htm as a testcase I make all checksums using selenium in dictionary.txt ('show_all' : ' 8548bccac94e35d9836f1fec0da8115c, 'hide_all' :3fdec30c2731d22e2516b1cd1261a1e1, 'filter_by_id_click' : 3fdec30c2731d22e2516b1cd1261a1e1) and so on..
  • The use cases are done in selenium (driver.findbuttonShowAll.click(), assert(md5(html)==dict['show_all])
  • Further development that doesn't brake expected html output is safe, when assertion fails I diff the htmls...

UPDATE : Note that taking this approach can check dynamic behaviour, because of the fact that the md5 strings are build browserspecific from html in memory. The html is extracted from a selenium webelement that in its turn is taken from the current webdriver state.

I am not changing the way selenium handles things, just comparing the underlaying html.

To get dynamic content, I pass the loaded webdriver to a helper method that extracts the outerHTML of a given xpath expression (python)

def get_outerHTML_by_xpath(p_driver, p_xpath_expression):
    webelement = p_driver.find_element_by_xpath(p_xpath_expression)    
    outerHTML= p_driver.execute_script("return arguments[0].outerHTML" , webelement)    
    return outerHTML

Peter, welcome to SQA. Rather than label the MD5 approach bad or not bad, it may help to think about how it relates to an image capture approach, i.e. an approach in which you compare browser screenshots. I do not mean to imply that these two approaches are the only ones worth considering; rather, they are worth comparing because both ignore the meaning of the page in exchange for a easily described, coarse-grained comparison.

  • External resources. The MD5 approach, as you stated it, ignores external resources such as stylesheets, images, Javascript, and applets, whereas the image approach captures the visual gestalt of everything loaded in the browser. Of course, with some extra work the MD5 approach could checksum those external resources, too.
  • Hidden fields. The MD5 approach includes hidden fields in the checksum; the image capture approach does not. Hidden fields may be irrelevant to whether the page is visually correct, but knowing their values may help diagnose problems with how the page (or subsequent pages) behave. If a hidden field is dynamically generated, the MD5 approach may fail.
  • Dynamic IDs. Some frameworks use dynamic IDs in the HTML. The MD5 approach fails with dynamic IDs; the image capture approach is oblivious to dynamic IDs.
  • Other dynamic behavior. Some user interfaces incorporate the current time into the HTML, either visually or in hidden fields. Both MD5 and image capture approaches would fail with that kind of page.
  • Browser dependencies. The same HTML (and external resources) may behave correctly in one browser but not in another. Of course behavior can vary among different versions of the same browser, too. The MD5 approach would tell you whether the right HTML is loaded into the page. Different browsers will not necessarily load resources in the same order (or load them at all), so your MD5 approach would need to deal with that. The image capture approach would require capturing images per browser, which would increase your maintenance cost.

Of course there are other ways to approach visual capture -- see Sikuli for example -- but that is beside the point, which is this: there is more than one way to approach coarse-grained comparisons, and each has their own trade-offs.

Another way to think about your question is in terms of how it would fit into your testing process. Somewhere in the process, someone tests the user interface manually to look for things that are hard to detect automatically, e.g. visual/layout problems, timing issues, and spelling/wording problems. Would that be the same person who maintains the automated tests? If not, how do you avoid duplicating work? Of course this problem arises with assertion-style automation too, but assertion-style testing does not attempt to account for visual issues, whereas you could argue that part of deciding that it is time to checksum a page is that the page is visually correct.

  • 1
    +1 and thanks a lot for the extended answer,... but I think you are overlooking the fact that I get the html string in memory not the one in the initial repsone . That way hidden fields (display=none for example), dynamic behaviour and even browser dependencies are taken in account. The checksums are made and tested by driver.firefox, driver.IE and driver.Chrome for example... The whole purpose of my testing is testing dynamic behaviour and does that perfectly fine.
    – Peter
    Oct 2 '12 at 19:39
  • As far as dynamic ids and other dynamic go : indeed, that's right. I even had to deal with them, but so far I could easily change dynamic ids with deterministic ones (that's always a better approach if possible imo), and in the worst case I can leave dynamic parts out of the comparison string. I forget to react on extarnal resources : they too are compared as long as they change the DOM (css, javascript,...). Of course I cannot notice if an image is replaced...
    – Peter
    Oct 2 '12 at 19:46
  • "I get the html string in memory not the one in the initial [response]". Can you elaborate on that? Are you actually fetching the HTML from the browser after the page is loaded?
    – user246
    Oct 2 '12 at 19:49
  • I updated my question with a code fragment that shows how I read the dynamic html. (in comment code is less readable)
    – Peter
    Oct 2 '12 at 19:58
  • If you are pulling the HTML at runtime and hashing that, what are you comparing it to? In your original message I assumed you were storing a baseline somewhere to then compare against what you get in subsequent runs. By one of your comments above, it looks like you might be getting it for each browser and comparing those? Can you clarify?
    – Sam Woods
    Oct 2 '12 at 23:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.