Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I suppose there is exception handling available in all high level language. And I often find test method written as -

public void testLogin()
 try{
     testCtrl.type("loginText", "loginName");
     testCtrl.type("loginPasssword", "password");
     testCtrl.clickAndWait("loginButton");
     AssertText("WelcomeUser");
 } catch(Exceltion e) {
    print(e.stackTrace);
    fail();
 }

Now suppose password field is not available then contrl would come to catch block and would print the stack trace, followed by test failure. Now why to do this? why not just let test fail as soon as password field is not found. I can not figure out the objective of doing try catch with such method during test automation. Do you also follow this "obtuse magical" approach with your test automation?

share|improve this question

7 Answers 7

up vote 4 down vote accepted

I have to admit that I probably do this too much, however, not at the test level. I typically add try catches to each of the actions that my tests call as there is often times where although it is important for me to see if a certain action did not execute the way that was intented, it is also important for me to see if anything happened after that point that should not have. This allows us the possibility to find more than 1 bug per test.

For example, in your example, what if the element locators were wrong for both the password field and the login button were wrong/had changed. The test would be run twice before this was discovered. I'm not certain if this is what you're actually looking for, however, here's how I would have written your test method in C#. (sorry for the length, I figured it'd be best just to include everything)

namespace example
{
    public static class testData
    {
        public static bool fail;
        public static StringBuilder failMessage = new StringBuilder();
    }

    public class tryCatchExample
    {
        public static IWebDriver _browser = null;
        public static IWebDriver browser
        {
            get
            {
                if (_browser == null)
                    _browser = new InternetExplorerDriver();
                return _browser;
            }
        }

        [SetUp]
        public static void SetUp()
        {
            testData.fail = false;
            browser.Navigate().GoToUrl("urlOfLoginPageUnderTest");
        }

        [TearDown]
        public static void tearDown()
        {
            if (testData.failMessage != null)
            {
                Console.WriteLine(testData.failMessage);
                testData.failMessage.Clear();
            }
            _browser = null;
        }

        [Test]
        public static void testLogin()
        {
            actions.enterText("loginText", "loginName", browser);
            actions.enterText("loginPassword", "password", browser);
            browser.FindElement(By.Id("loginButton")).Click();
            //The below assert is optional as you could also continue to see if you are able to login without credentials
            Assert.AreEqual(testData.fail, false, "We are not able to continue as not all requirements to login have been met.");
            actions.assertTextPresent("WelcomeUser", browser);
            Assert.AreEqual(testData.fail, false);
        }
    }
    public static class actions
    {
        public static void enterText(string text, string idOfElement, IWebDriver browser)
        {
            try
            {
                IWebElement elementToTypeIn = browser.FindElement(By.Id(idOfElement));
                try
                {
                    elementToTypeIn.SendKeys(text);
                }
                catch
                {
                    testData.failMessage.Append("Was not able to enter text \"" + text + "\" into element with id of " + 
                        idOfElement + Environment.NewLine);
                }
            }
            catch
            {
                testData.failMessage.Append("Could not find element with id of: " + idOfElement + Environment.NewLine);
                testData.fail = true;
            }
        }

        public static void assertTextPresent(string textToFind, IWebDriver browser)
        {
            try
            {
                Assert.IsTrue(browser.FindElement(By.TagName("body")).Text.Contains(textToFind));
            }
            catch
            {
                testData.failMessage.Append("Was not able to find text: " + textToFind + Environment.NewLine);
                testData.fail = true;
            }
        }
    }
}
share|improve this answer
    
Very well explained Lyndon –  Tarun Jun 6 '11 at 11:09
    
Hey Lyndon, I liked the idea of testData.failMessage, this is some thing which I could use to print in test report to see all the failures test has come across while trying to login. While reading your test method "testLogin" I feel that we should not keep more than one assertions in tests. Since when any of the assertion fails then rests are not going to be checked any way. I suppose this is the reason you mentioned a comment stating - "//The below assert is optional as you could also continue to see if you are able to login without credentials" right? –  Tarun Jun 7 '11 at 4:13
    
That is correct. –  Lyndon Vrooman Jun 7 '11 at 9:16

The exception handling should not be done at the test level (with exceptions) for two reasons that I can think of:

  1. Certain times you want the test to fail rather than handle the exception for the reporting purpose. Exception handling will tell the automation developer that there was an exception/error, but will show the test as pass, which may not be true.
  2. The exception described above is technically not the exception in the test.

This is one of the benefits of using the Page Object model. The login page class can have the method to type in the password field which can have the exception handling for existence of the password field.

An example in Java:


public class LoginPage  
{  
public typePassword(String pwd)  

      if(!passwordField.exists())  
      log.error("Password field does not exist");  

      passwordField.type(pwd);  
}  

The test method can call this method, and does not need to have an exception handling in the test method itself.

share|improve this answer
    
is if block really needed here? if password field were not available then your harness would log an exception any way, is not it? –  Tarun Jun 6 '11 at 3:50
    
Depends on what you want your block to do. If you want that the test should stop here, and exception be thrown, then what you mention is correct. In few cases, you might just want to skip that step and log an error, then the if block would work better. –  Suchit Parikh Jun 6 '11 at 17:20

For the simple test method above, any failure can be understood by the tester easily without the help of printstacktrace.

There are some failures that are hard to debug. For example, suppose we pass three parameters to a test method. We failed to implement session maintenance for one of the 3 objects passed as parameters and it happens to be passed as null as a consequence. Such failures are hard to debug. It can be easily narrowed down to the root cause using stacktrace.

For example let's take a sample test case to first login into facebook and then add a friend. Suppose login code is written as a separate file for re-usability under the same namespace. First I call the login module in the test case and then call the add-friend module. When I run the test case the user logs into facebook, and fails at the add-friend module call because I missed to maintain the session of the selenium object. When the call is returned from the login module, it doesn't know that the user has already logged in since the selenium object is null. When I try to add-friend it throws an exception. The root cause of this exception can be traced more easily with printstacktrace.

/Test Case Code/


package resources.module;

public class addFriend { //To set the baseURL and server port DefaultSelenium sel = new DefaultSelenium("localhost", 4444, "*firefox","http://facebook.com/");

//Declaration to instantiate class for reusable pieces of code written as module in /resources/modul resources.module.loginFriend loginFacebook= new resources.module.loginFriend(); resources.module.addFriend addFbfriend = new resources.module.addFriend();

public void setUp() { ...Starting server code }

@Test public void testaddFriend(DefaultSelenium sel, String user1, String pass1, String user2, String pass2) { loginFacebook(sel, user1, pass1); //User1 logs into facebook by calling login module addFbfriend(sel, user1, pass1, user2, pass2) //User1 adds User2 as friend }

Now lets see the login module code package resources.module;

public class loginFriend { @Test public void loginFb(DefaultSelenium sel, String username, String password) { Code to login } }

Now lets see the addfriend module which throws exception

package resources.module;

public class addFriend { public void addFbfriend(DefaultSelenium sel, String user1, String pass1, String user2, String pass2) { sel.click("link=Profile"); //throws exception since selenium object is passed as null } }

share|improve this answer
    
I wish you had written pseudo code for this, can you? –  Tarun Jun 6 '11 at 3:51
    
Sorry for that, added pseudo-code –  Aruna Jun 6 '11 at 22:05

I think this depends on what you're testing to a certain degree. There are some cases where an exception being thrown is the pass criteria.

For example, I write tests to verify that an API upholds the rules of an expired trial; changing files and / or saving them is not allowed. I'll use exception handling in this case to catch the error coming out of the API and set a boolean that I then evaluate. If the exception is thrown and the data isn't changed, the test is a pass.

However, I agree with the other posts that in most other cases, you're probably better off using a test framework that will handle these and report them as test failures.

share|improve this answer

Most test runners reports three kinds of test results:

  • E(rror): An unhandled exception occured
  • F(ailure): A test assertion failed
  • .(everything OK)

So no, you should not do exception handling in the test methods.

share|improve this answer
    
very precisely said –  Tarun Jun 6 '11 at 16:06

It's better to distinguish between test failure and test error; i.e. when the password field is not available, it's not a failure of Login feature itself, but instead is an error in the test setup/execution.

Thus, I prefer to not catch such exception to see if my tests are outdated or the SUT is completely broken. It's especially helpfull when I have a lot of tests in the test run: many test errors could mean that something is wrong, while many test failures may cause whole team to panic.

share|improve this answer

A decent test framework will take care of catching and reporting exceptions for you. I don't catch the exception explicitly unless I need to add something to the report that clarifies the exception's context, e.g. values of variables or database IDs.

share|improve this answer
1  
I agree, your test framework/runner should deal with catching exceptions. If you feel that you do need to catch an exception then you should catch the exact type, not a general exception in case you catch something that you shouldn't have. –  stuartf Jun 6 '11 at 10:20
1  
I think that's true to the same extent that it's true to programming in general (i.e. beyond writing tests). –  user246 Jun 6 '11 at 17:42

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.