Tag Info

Hot answers tagged

10

Solution Overview To solve this problem, you will want to intelligently select a manageable set of combinations based on a pairwise coverage approach (explained below) or a more thorough variation of combinatorial test design. Glowcoder and user246 have good points. I particularly like testerab's comment for reasons that will become clear in a minute. ...


9

I know this problem way too well. There's no "right" answer, unfortunately, but there are some things you can do to help with this problem. Dependency map - do you have a list of application features that have heavy dependencies and tend to break when changes occur in other areas? If you know changes in feature X tend to break feature Y, you know you ...


7

Actually, the underlying heuristic for combinatorial testing of multiple input parameters is that they are interdependent and the specific values assigned affect on a common output condition or state. Based on the concept of testing various combinations of input variables that affect a common output condition or state, combinatorial testing may not be ...


6

I always like to get code coverage for my functional tests, but not because I want to hit a certain percentage of code coverage. I like it because: It points me to areas of the code that are not covered. There are areas of the code that are very difficult to unit/integration test without having the entire system in place and doing end to end tests, so I ...


6

This is a tool-agnostic question. All of them work the same way: they instrument the code of your system. Each time a line or branch of your code is visited at runtime, the tool caches this information. Afterall, the number of visited lines (or branches) is divided by the total number of lines (or branches). This is done in the scope of a single class/file, ...


5

I wouldn't necessarily put it high on the ToDo list, but I think it's beneficial to measure test code coverage to find dead tests. You probably won't get to 100%**, but you can find dead functions and binaries - which makes a big difference when you have a 20-hour automation run you're trying to whittle down to an overnight run. ** note - test code often ...


4

Remove the uncovered line from your .py file and use python -c "import foo; foo.main(args);" You can wrap that in an alias or a bash function.


4

Do all combinations need to be run? This sounds like a classic place to apply all-pairs testing. I haven't played with this yet myself, but I hear good things about Hexawise which might help with display and analysis of the results.


4

How to do it Let's let your dimensions' sizes be C1, C2 ... Cn where n is the number of dimensions. So, C1 might be 3 if your values are Windows, Mac, Linux (I'm sure you'd have different versions of Windows and what not, but for the example, it works.) Your total number of tests will be C1 * C2 * ... * Cn. I'm sure you already have a 2d matrix defined ...


4

Firstly, I've been exactly where you are and I know the pain you're going through. It's also very 'cool' at the moment for everyone to talk about 'Automation' and how it's a 'golden ticket' to Continuous Integration. Everybody wants their stuff out fast. There are ways to tackle this, but personally, I think a lot of comes with having an understanding of ...


3

Not exactly. Unit testing only isolates units in isolation. In these tests all dependencies to other units are mocked or stubbed out. So how do you know those units together do what they are supposed to do? Code tends to grow hierarchically in complexity, and with that growth comes more and more units working together, more and more groups of units ...


3

The short answer is situations when a bug can occur only when three or more states must be set a certain way. The long answer ... Pairwise testing good technique that actually applies some pretty sound logic. That said, it is fairly easy to understand what the potential tradeoffs are .. let me illustrate using an example I have used previously. Let’s ...


3

The simplest option is to just mark the line as ignored by your coverage tests. You know more than coverage.py does, you can just excuse the line from the measurements: if __name__ == '__main__': # pragma: no cover return main(sys.argv) You can also use some tricks with coverage.py to get it to measure code in launched subprocesses. This sounds ...


3

The underlying assumption for those methods is that any two parameters are independent from each other, so any two parameters that do have some kind of dependency will show poor results, i.e. your coverage will not be as good as expected. When I tried to use pairwise the first step was to find independent enough parameters, I couldn't find any that will also ...


3

There are many cases where unit testing as you have described would not be "sufficient". (And you haven't really defined what you mean by "sufficient" in this case. Good enough to move the code to Production? Good enough to pass it on to QA? Good enough to please your boss? Good enough to feel like you did a good job? Something else?) In most practical ...


2

In a sense, this is like the UI automation problem. A human is the best judge of quality, and while an algorithm can't always tell you the UI is right, it might be able to tell you the UI is wrong and it might be able to tell you if something has changed since last time. As I understand it, you have two problems: an image comparison problem and a ...


2

You need to reduce your 11- or 12-dimensional cross product into a manageable number of combinations. I have had some luck using all-pairs (see http://en.wikipedia.org/wiki/All-pairs_testing) testing. Once you reduce your test space using all-pairs, you ought to be able report the results in a spreadsheet.


2

Similar question posted on Software Testing Club Robotium looks like it help fulfill some of your needs? (props to Stephen Janaway)


2

Try these articles about RCov and simple_cov http://highgroove.com/articles/2011/03/01/code-coverage-and-ruby-1-9.html http://www.storm-consultancy.com/blog/development/tools-plugins/generating-code-coverage-metrics-for-a-ruby-on-rails-project-with-simplecov/ and rails_code_qa uses both of the above so also worth a look ...


2

For backend coverage (mostly java), we use Cobertura: http://cobertura.github.io/cobertura/ The developers tend to use Emma, in Eclipse: http://emma.sourceforge.net/ For frontend (JavaScript), I've heard good things about ScriptCover: http://googletesting.blogspot.com/2011/10/scriptcover-makes-javascript-coverage.html Good luck!


2

Have you tried the book The Art of Unit Testing ? http://www.manning.com/osherove/


2

Coverage is always coverage related to some model. This often gets skipped over, which leads to much confusion "you said you had 100% coverage so how come there's a bug?!" When you're looking at unit tests, then it's possible to use code coverage as an indicator (there are tools that can measure what percentage of the lines in your code are exercised when ...


2

I have encountered exactly the same problem and there is no "right" answer, but I can share with you what I've learned: 1) Code changes in one place can cause problems in another. Unless your project is so incredibly well documented that you know exactly what a change can impact, testing the full application on releases is important. 2) Find a balance ...


2

You have many excellent answers already. Adding my 2 cents to say that your story hits close to home. Management and I have come to an agreeable compromise which is that I created "Mini regression" test scenarios for only the high traffic areas and modules of the program. The Mini Regression is performed as needed, but usually during installation testing. ...


1

Yes, we do just that. We use Python and Coverage. I am curious what is business reason to eliminate unit tests. Unit tests are much closer to code and if any fails, finding and fixing the bug is much easier. The only reason to eliminate unit test is if another unit test covered the same execution path. OK, after comment from OP: yes, it is valid reason to ...


1

Can't help with .net as I mainly have experience at a java shop. We use a combo of sonar and Jacoco. Check out Jacoco! I just installed it on the machines where we execute our service tests and the reports it generates are nice and can be integrated into CI easily (using ant). It's actively worked on, and has a Maven plug-in. The coolest feature for me ...


1

Visual Studio 2012 has a pretty good code coverage tool. I have used it to measure code coverage of a web service. It is easy to integrate into build systems, you have out of the box support for TeamBuild (the TFS build) - see a more detailed article here. According to msdn it has result merging. Code coverage is also integrated in Microsoft Test ...


1

If price is an issue, you can use ncover 1.5, which is still open source on sourceforge. Another option is coverity test advisor.


1

What you really want to measure with functional tests is functional coverage: how much of the functionality of the program was tested? Unfortunately, that's hard to measure in an automated fashion; the best measurement we have is by hand, correlating tests to requirements and counting up what didn't get covered. Code coverage can be used as an indicator of ...


1

Personally I don't run code coverage on test code, as a tester with limited time, I simply have too many other things that need my limited time and attention. What I do run though is static code analysis with Visual Studio or fxCop as a way to keep test code quality high. I actually joke that visual studio Pro should be called Visual Studio un-Professional ...



Only top voted, non community-wiki answers of a minimum length are eligible