Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Disclaimer: I've never used any test tools before, as I'm in the developer team, not the QA team.

Our team is tasked with the ultimate goal of writing an automated load test script, which will be hitting on our JS-based web application. We will most likely be using JMeter.

As we're taking it one step at a time, we started with writing test scripts using Selenium IDE with Firefox driver, then migrate to HtmlUnitDriver as we want to eventually run multiple instance of the tests at the same time. We are running selenium-server-standalone-2.41.0.jar in the background. The Selenium IDE has WebDriver playback enabled, and is using "selenium-server-standalone-2.41.0.jar", "htmlunit-2.13.jar", and "htmlunit-core-js-2.13.jar" as the IDE extensions.

The test will load our login page (which, unfortunately, contains <iframe>), login to our dashboard page, check for existing text, and logout from that page.

Here is a simplified portion of the login page:

<html>
<head><title>footest</title></head>
<body>
    <div id="container" align="center">
        <div id="login">
            <div id="login-panel">

                <iframe src="http://localhost/auth/form">

                    <div id='login-form'>

                        <form action="http://localhost/auth/form" method="post">
                        <div class='row'>
                            <label for='username' class='vertical'>Username</label>
                            <input class="text" id="username" name="username" type="text" value="" />
                        </div>

                        <div class='row'>
                            <label for='password' class='vertical'>Password</label>
                            <input class="text" id="password" name="password" type="password" value="" />
                        </div>

                        <div class='row'>
                            <input class="float-right" id="login" name="login" type="submit" value="Log In" />
                        </div>
                        </form>

                    </div>

                </iframe>

            </div><!-- login-panel -->
        </div><!-- login -->  
    </div><!-- container -->
</body>
</html>

Here are the steps we recorded with Selenium IDE:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head profile="http://selenium-ide.openqa.org/profiles/test-case">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<link rel="selenium.base" href="http://localhost/dashboards/" />
<title>footest</title>
</head>
<body>
<table cellpadding="1" cellspacing="1" border="1">
<thead>
<tr><td rowspan="1" colspan="3">footest</td></tr>
</thead><tbody>
<tr>
    <td>open</td>
    <td>http://localhost/dashboards?back=http%3A//localhost/dashboards/main.html</td>
    <td></td>
</tr>
<tr>
    <td>waitForPageToLoad</td>
    <td>10000</td>
    <td></td>
</tr>
<tr>
    <td>selectFrame</td>
    <td>index=0</td>
    <td></td>
</tr>
<tr>
    <td>waitForText</td>
    <td>//label</td>
    <td>Username</td>
</tr>
<tr>
    <td>type</td>
    <td>//input[@id='username']</td>
    <td>foo</td>
</tr>
<tr>
    <td>typeAndWait</td>
    <td>//input[@id='password']</td>
    <td>bar1234</td>
</tr>
<tr>
    <td>clickAndWait</td>
    <td>//input[@id='login']</td>
    <td></td>
</tr>
<tr>
    <td>waitForPageToLoad</td>
    <td>20000</td>
    <td></td>
</tr>
<tr>
    <td>pause</td>
    <td>2000</td>
    <td></td>
</tr>
<tr>
    <td>waitForText</td>
    <td>id=graph-table</td>
    <td>No records found</td>
</tr>
<tr>
    <td>click</td>
    <td>//a[@id='logout']</td>
    <td></td>
</tr>
</tbody></table>
</body>
</html>

When we run the tests with WebDriver playback + firefox browser, we are able to run the test successfully. However, when we switch to using htmlunit, we were not able to run it. The error is as follows:

17:28:25.432 INFO - Command request: getText[//label, ] on session 87338f89-7d2f-4387-b85e-3685a1a8a2ac
17:28:27.135 INFO - Got result: ERROR,Element //label not found on session 87338f89-7d2f-4387-b85e-3685a1a8a2ac

Similar error occured on selecting the input field with id='username', or clicking on the logout button at the end.

The questions are:

  1. How can we reliably run the scripts without spawning a browser?
  2. Are we going in the right direction with using HtmlUnit?
  3. Any other tools / hints / etc. we should consider to test this without browser?
share|improve this question
    
Why are you going this route if you want to do load testing? Why not dive straight into JMeter? Selenium is great for functional tests but not the tool of choice for performance –  Phil Kirkham Apr 18 at 12:55
    
We had functional test scripts written in Selenium (by the QA team), which we thought might be able to integrate to load test. Also, when we looked at JMeter initially, we got the impression that it does not fully support testing ajax-based / scripted websites. That's why we thought of creating scripts in Selenium and then convert it to be executed by JMeter. –  Sutandiono Apr 19 at 16:06
    
add comment

2 Answers 2

up vote 1 down vote accepted
  1. Selenium tests are always run against a browser. HTMLUnit mimics a browser in a headless form.

  2. Running tests against a headless browser is great for faster feedback during development, but in the long run you will want to run your tests against real browsers to find browser compatibility issues.

  3. Did you try PhantomJS, its a headless webkit browser. I know of teams that run the tests vs PhantomJS first locally, before they fire it up against a grid like Saucelabs to test all different browsers. If you Google on PhantomJS and Webdriver you will find a lot of people playing with it, like this blog post: http://itkosmopolit.wordpress.com/2013/11/23/quick-start-of-running-selenium-webdriver-against-phantomjs-in-java-on-windows/. As it is a "real" browser you can also take screenshots in a headless mode. :)

share|improve this answer
    
Thanks for the suggestion on PhantomJS. It looks promising. We'll check it out and hopefully can get something out of it. –  Sutandiono Apr 18 at 8:33
    
We checked out the example from IT-Kosmopolit and were able to run some basic test scripts. However now we are confused: Should we write scripts in JS per the example in the site, as well as these (1) people (2), or do we keep writing it in Java and just use PhantomJS as the driver? –  Sutandiono Apr 21 at 3:19
    
I would stick with what you know, keep using Java and PhantomJS as a browser which supports webdriver. Found another post about using Java and PhantomJS appneta.com/blog/automated-testing-java –  Niels van Reijmersdal Apr 21 at 9:36
    
We continued to use Java, which is like what you suggested. We still couldn't figure out how to select certain elements such as <iframe>, and are looking into how to take screenshots of steps in our tests to see whether we are doing things correctly. Now what we need is to brush up our Java and webdriver skills. –  Sutandiono Apr 21 at 9:43
    
Bit of topic, but Selenium handles iFrames as an window which you have to switchTo(). See stackoverflow.com/questions/9942928/… . Goodluck with learning! :) –  Niels van Reijmersdal Apr 21 at 9:59
show 2 more comments

Selenium IDE is not a reliable way to create tests. It would be a maintenance nightmare. You might be better of using the webdriver package as such. It supports multiple programming languages. If you know Java, Python or C#, you should be able to pick it up in a day. Most QA teams forget that test automation is software development after all. Let me know if you do face any specific issues.

share|improve this answer
    
We sort of noticed that, too. However, since we have limited Java knowledge and there are not too many examples using Python, we thought of using Selenium IDE as a base and expand from there. Most examples we saw show us how to perform search with Google. There's not many on selecting iFrame and JS-generated texts. But thanks for the input. We'll try to look into it. –  Sutandiono Apr 18 at 1:47
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.