4

The team I am currently working with has a very large application to test (several custom programs, running in a distributed environment), and has built up a very large set of automated test cases for regression and feature testing. These tests are large and there are a lot, so full test runs are dispatched across many machines, the results gathered, and then imported into a custom web app.

There are some very nice tools out there for writing the automated tests, but there doesn't seem to be much for managing a "automated testing workplace". We ended up writing custom applications to do all of the following:

  • Schedule tests to run on one of several machines
  • Gather results, import into database
  • Web application for result viewing and querying
    • Results per run/software version/test machine/etc
  • Integration with external bug tracker for tracing failures to tests
  • Nice to have: Smart analysis of results
    • "This test started failing/passing"
    • "This test is unreliable/intermittent"
    • etc

Does anyone know of anything off the shelf that handles all of the above? Most of what I have found before is very focused on manual testing, which we don't do much of.

Improve this question
1
  • 1
    At my company we have also created custom solutions for much of this. I would love to see a response from someone with some tools that have "everything" you would need, but I don't know of any. It seems like a pretty big gap in the industry if you ask me.
    – Sam Woods
    Apr 16, 2013 at 18:02

1 Answer 1

Reset to default
3

If I'm understanding correctly you could use a Continuous Integration tool like:

Jenkins is an award-winning application that monitors executions of repeated jobs, such as building a software project or jobs run by cron.

TeamCity is a user-friendly continuous integration (CI) server for professional developers and build engineers, like ourselves. It is trivial to setup, manage and use. It scales perfectly for a company or a team of any size.

Both have numerous features, which you can browse via their respective websites. We are currently using Jenkins to run our automated tests (smoke, regression and unit tests). It does a nice job of reporting and is customizable so that you can a variety of plugins (which are available via the Jenkins site).

We push tests to different machines and test results for the tests are reported within Jenkins.

Improve this answer
3
  • We currently use Jenkins for running builds and unit tests, but not for larger system tests. I don't feel like the reporting in Jenkins is useful enough for managing 1000's of granular test results in a queryable way.
    – Chris Pitman
    Apr 15, 2013 at 20:48
  • 1
    As an example, we have 1000's of long running system tests. I would want to kick those off periodically, but also be able to kick off each test manually. In Jenkins, every test would have to be a separate task so that I can dispatch granularly... but if I then want to run all tests with a tag, then what? Click each task to dispatch it?
    – Chris Pitman
    Apr 15, 2013 at 20:59
  • Good point Chris. I agree with downside of Jenkins. To combat that - we have tests broken into their individual state and then have them packaged into suites so that you can either run them separately or run them as a suite.
    – aftershock9
    Apr 16, 2013 at 5:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.