Tag Info

Solution Overview To solve this problem, you will want to intelligently select a manageable set of combinations based on a pairwise coverage approach (explained below) or a more thorough variation of combinatorial test design. Glowcoder and user246 have good points. I particularly like testerab's comment for reasons that will become clear in a minute. ...

Actually, the underlying heuristic for combinatorial testing of multiple input parameters is that they are interdependent and the specific values assigned affect on a common output condition or state. Based on the concept of testing various combinations of input variables that affect a common output condition or state, combinatorial testing may not be ...

This is a tool-agnostic question. All of them work the same way: they instrument the code of your system. Each time a line or branch of your code is visited at runtime, the tool caches this information. Afterall, the number of visited lines (or branches) is divided by the total number of lines (or branches). This is done in the scope of a single class/file, ...

I wouldn't necessarily put it high on the ToDo list, but I think it's beneficial to measure test code coverage to find dead tests. You probably won't get to 100%**, but you can find dead functions and binaries - which makes a big difference when you have a 20-hour automation run you're trying to whittle down to an overnight run. ** note - test code often ...

The short answer is situations when a bug can occur only when three or more states must be set a certain way. The long answer ... Pairwise testing good technique that actually applies some pretty sound logic. That said, it is fairly easy to understand what the potential tradeoffs are .. let me illustrate using an example I have used previously. Let’s ...

How to do it Let's let your dimensions' sizes be C1, C2 ... Cn where n is the number of dimensions. So, C1 might be 3 if your values are Windows, Mac, Linux (I'm sure you'd have different versions of Windows and what not, but for the example, it works.) Your total number of tests will be C1 * C2 * ... * Cn. I'm sure you already have a 2d matrix defined ...

The underlying assumption for those methods is that any two parameters are independent from each other, so any two parameters that do have some kind of dependency will show poor results, i.e. your coverage will not be as good as expected. When I tried to use pairwise the first step was to find independent enough parameters, I couldn't find any that will also ...

In a sense, this is like the UI automation problem. A human is the best judge of quality, and while an algorithm can't always tell you the UI is right, it might be able to tell you the UI is wrong and it might be able to tell you if something has changed since last time. As I understand it, you have two problems: an image comparison problem and a ...

You need to reduce your 11- or 12-dimensional cross product into a manageable number of combinations. I have had some luck using all-pairs (see http://en.wikipedia.org/wiki/All-pairs_testing) testing. Once you reduce your test space using all-pairs, you ought to be able report the results in a spreadsheet.

The simplest option is to just mark the line as ignored by your coverage tests. You know more than coverage.py does, you can just excuse the line from the measurements: if __name__ == '__main__': # pragma: no cover return main(sys.argv) You can also use some tricks with coverage.py to get it to measure code in launched subprocesses. This sounds ...

Try these articles about RCov and simple_cov http://highgroove.com/articles/2011/03/01/code-coverage-and-ruby-1-9.html http://www.storm-consultancy.com/blog/development/tools-plugins/generating-code-coverage-metrics-for-a-ruby-on-rails-project-with-simplecov/ and rails_code_qa uses both of the above so also worth a look ...

Similar question posted on Software Testing Club Robotium looks like it help fulfill some of your needs? (props to Stephen Janaway)

Can't help with .net as I mainly have experience at a java shop. We use a combo of sonar and Jacoco. Check out Jacoco! I just installed it on the machines where we execute our service tests and the reports it generates are nice and can be integrated into CI easily (using ant). It's actively worked on, and has a Maven plug-in. The coolest feature for me ...

Visual Studio 2012 has a pretty good code coverage tool. I have used it to measure code coverage of a web service. It is easy to integrate into build systems, you have out of the box support for TeamBuild (the TFS build) - see a more detailed article here. According to msdn it has result merging. Code coverage is also integrated in Microsoft Test ...

I have tested Java enterprise systems for about 10 years, both web and desktop client. Each of the systems I tested had a different requirements model, but the evaluation of coverage metrics (as it applies to answering if the product is ready to ship) was always a major concern. The most effective coverage evaluation in our experience was a two-pronged ...

Personally I don't run code coverage on test code, as a tester with limited time, I simply have too many other things that need my limited time and attention. What I do run though is static code analysis with Visual Studio or fxCop as a way to keep test code quality high. I actually joke that visual studio Pro should be called Visual Studio un-Professional ...

I'm with Alan on this one - a lot of test automation code will cover error handling which may not get exercised often - and if you're really lucky, it won't ever get exercised (you do include error handling for every routine, right?) Also, if you have a lot of libraried routines, you might find that one set of tests uses a relatively small subset of your ...

General Rules: At least one test method should be written for every public service method and every new or overridden repository method Where applicable, positive and negative tests should be written Every Assert should include a relevant, meaningful error message for when failure occurs The Arrange, Act, Assert pattern should always be followed (see ...

I highly recommend the book "Pragmatic Unit Testing in C# with NUnit" by Andrew Hunt and David Thomas. If you are using Visual Studio as your IDE you might also want to check out MSDN (e.g. http://msdn.microsoft.com/en-us/library/ms182532.aspx)