5

What tools exist to automatically generate test scripts (or test script templates) from a formal specification or a program's source code?

EDIT:
The background of this question is, that I’m occasionally confronted with legacy web applications as both, a developer and a QA person. The applications are written in either Java or a combination of Java and PL/SQL, and need to be maintained (maybe sometimes rather partially refactored in the sense of re-architectured and rewritten) and amended with new features. Usually there hasn’t been any test driven development, though unit tests were added later on, which test some important spots of the code and are run during nightly builds. Once in a while I get rotated out of one of those projects to face the next, depending on workload, schedules, business opportunities etc. I’m looking for enhancements of my testing toolset, to both being able to quickly fill testing gaps for those kind of applications and to improve the long term stewardship (rewriting/migration support).

Test scripts would be the executable test, which can aggregated to test suites, test script templates would be the test scripts with placeholders for test data, which get replaced somehow to make them complete test scripts (either preprocessed or during execution).

Since those are applications that already are live, you indeed would be able to record the outcome of the tests as is and store it as expected result for a given typical input.

I also found a web source that talks about the extraction of models from source code http://drops.dagstuhl.de/opus/volltexte/2007/881/, so maybe it would be possible to put a model on top? And generate test scripts from that model?

I would like to know, if somebody used a tool following the ideas described above, and how it turned out.

Improve this question
11
  • By test scripts you mean scripts for manual or automated tests?
    – dzieciou
    Dec 23, 2012 at 6:50
  • @dzieciou Definitely for automated tests. Is there a more unambiguous term than test scripts in this context?
    – stacky-bit
    Dec 23, 2012 at 11:34
  • 1
    You've received a wide spectrum of answers, one of them using a wording: "maybe this is what you're looking for" (see Dmitry's answer). This is because your question is generic and can be interpreted in different ways. Which of this answers matches the problem you want to solve? What is the specific problem you want to solve? So far, I'm confused what you are asking for.
    – dzieciou
    Dec 24, 2012 at 8:29
  • 2
    I'm afraid you will find nothing to automatically generate complete unit tests from source code: defining expected test result requires brain.
    – dzieciou
    Dec 24, 2012 at 12:07
  • 1
    It might be more effective for you to focus on the problem and an ultimate goal you want to achieve, instead of researching solutions you have in mind. There are many questions about similar problems here: sqa.stackexchange.com/questions/149/…, sqa.stackexchange.com/questions/2327/…, sqa.stackexchange.com/questions/996/….
    – dzieciou
    Dec 24, 2012 at 12:08

2 Answers 2

Reset to default
3

This is surely the low bar, but there are ways to test Java beans automatically; see for example http://macbeantechnology.co.uk/blog/automate-testing-javabeans/.

I think the academic literature has a lot to say about about generating tests from specifications. A Google search of "generate tests from formal specification" will turn up lots of results; see for example http://www.cs.gmu.edu/~offutt/rsrch/papers/scr-sp-JSTVR.pdf.

Someone on Stackoverflow recommended Agitar, a commercial product that purports to generate tests for Java code. I assume it starts with the (compiled) class files. Of course, when you generate tests from source code or compiled code, the resulting tests will not tell you anything about whether that particular implementation is correct.

Improve this answer
9
  • Ok, maybe "existence" of a tool is not really enough, and the scope of my question is a little too broad. I will have to edit it to be a little more specific. I especially would like to know about tools that you use on a regular basis or did succesfully use at least once, preferably non-homebrew ;-).
    – stacky-bit
    Dec 23, 2012 at 18:51
  • 1
    @stack-bit, there are tools for random or parametrized random test data generation. Searching online, on SE in general and here on SQA in particular will return you some results. Pure random data generators are used to check how your system behaves for invalid input or find security holes. However, even in those cases you need to use your brain to narrow/parametrize samples. In general, the generator cannot abstract the purpose of the application from the code.
    – dzieciou
    Dec 23, 2012 at 19:23
  • @dzieciou I don't want to touch on test data generation with this question. I consider test data generation as a different step. When I said script templates, that's what I had in mind: the test data will be provided by a different process to fill the holes.
    – stacky-bit
    Dec 23, 2012 at 23:30
  • @dzieciou Thanks for your pointer to stackoverflow and Agitar. Guess I should also look what's over there :-)
    – stacky-bit
    Dec 23, 2012 at 23:32
  • @stacky-bit: Ctrl+Shift+T in IntelliJ IDEA or in Eclipse will create unit test script template (aka stub) for selected methods of a component. Created test method will have empty body that you need to fill in. Is this want you're looking for? I'm sure you will find other widgets for generating test stubs when working with other types of interfaces, e.g., for HTTP or Web browser.
    – dzieciou
    Dec 24, 2012 at 8:19
3

stacky-bit, maybe Model-based testing approach is what you are looking for? Using this approach, you can generate all possible "abstract tests" that are based on the model. Such approach can be used on, for instance, military projects with a lot of people (such case is described in Experiences of Test Automation Dorothy Graham & Mark Fewster)

The pros of this approach is that you can generate tests automatically. In most cases, when you have a solid test-automation you will not be required to write the code, but just add new data into the Model and new tests will be automatically generated and the old tests will be improved.

The cons: it requires a lot of learning and knowledge. You will have to hire a consultant to start such project and teach the team or do a lot of reading and research by yourself. It still requires programming skills to implement low-level test automation layers.

Another way is using ATDD/BDD/Specification by Example approaches. Using such approaches and tools like Cucumber/RSpec you can create executable specifications. Those tools allows you to write the specification in Human (Domain/Business) languages and generate the stubs in the code. Then you will need to implement such stubs.

This option requires the programming skills, but makes the specification and the test code linked well.

Improve this answer
3
  • 1
    I'm afraid, I'm actually looking for too many different things simultaneously. I will have to rethink my question and amend it. Anyway, thank you for your answer so far. Model-based testing was certainly one of the things I had in mind. And ATDD/BDD was new to me, though I also thought of something in the line of Specification by Example. I have to do a little read up here.
    – stacky-bit
    Dec 23, 2012 at 23:23
  • Dmitry, have you any experience with Model-based testing? There are questions and answers about this approach here: sqa.stackexchange.com/questions/840/…
    – dzieciou
    Dec 24, 2012 at 12:11
  • @dzieciou - Alan's answer to the question you linked is spot on. I used model based testing a few times and it was successful when used for state based machines, but not so useful when applied to something like UI automation or rest web services. It can be used SOMETIMES with those features, if there is a specific flow or tree that you can follow and you want to test that.
    – Sam Woods
    Dec 26, 2012 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.