Is my test a perf unit functional smoke test or something else?

Question

Or how I learned to stop worrying and love the bomb.

Each test is different. Some are big and talk to databases, others quick and self contained.

We all know we'd prefer the fastest and most robust test that does the job.

But is there a point in trying to seperate tests into different boxes marked performance tests or functional tests? What does this gain us?

Answer 1

Categories can help. Categories can help with some things. You might use categories to help indicate:

• Who will write or run the tests. For example, unit tests are generally written by developers (and are sometimes called developer tests). Acceptance tests might be executed by testers or product owners.
• The characteristics being tested--e.g. functional tests or load tests.
• The purposes for which you might run the tests. You run smoke tests to help decide whether it would be worthwhile to run a more comprehensive set of tests. You run acceptance tests to help decide whether to accept the system.

In general, a categorization scheme is useful if it helps someone decide whether and how to think about or interact with the things being categorized. The ways we interact with tests are to write, read, run, or respond to them (and maybe some other stuff I've neglected). If a test categorization scheme helps you do those things, it's useful. Otherwise, it's not so useful.

Multiple dimensions, so multiple categorizations. One day for fun I analyzed a bunch of test categories to identify the dimensions or variables at play. I quickly identified thirteen dimensions (and of course there are others, but I stopped there). Each categorization scheme places any given test somewhere along one dimension, or maybe two.

But every test falls somewhere along each of the dimensions. So every categorization scheme tells you some (potentially) useful things about tests, but none tells you everything you might want to know about a test. A given test might be both a smoke test (which indicates a reason for running it) and a function test (which indicates what characteristic is being tested).

In the end, the value of a categorization scheme arises from your purposes for interacting with the tests, and the extent to which the scheme helps you interact well.

Answer 2

Yes, there is a point about separating performance from functional, "If it does not work for one, then it will never scale to many..." The skills needed to implement manual or automated functional tests and performance tests are very different from a tester perspective.

Performance testers have a very different view of the system and its architectural components than their functional brethren. Moreover performance testing typically involves a subset of business cases which produce the majority of the load on the system, typically in the range of 8-12 business processes per app/module. This contrasts to hundreds, or potentially thousands of functional test cases which much be confirmed before performance testing can even begin.

I have observed a lot of managers who don't understand the differences between the testing disciplines or the skill differences between the tester types try to lump everything together into one test. I have never observed a successful implementation of this model in almost two decades within the profession.

Answer 3

There are several reasons to organize related groups tests into different test suites.

• Purpose - the fundamental purpose of a functional test is different then the purpose of a performance test, or a stress test, etc., and thus provide us with different information about the product being tested.
• Scheduling - tests organized into suites enable the test team to schedule sets of tests on new builds. For example, we organize tests by groups such as Unit tests, BITS (similar to BVT), P1's, P2, Stress, Performance, Battery, etc. This allows us to prioritize which tests get ran firts on each new build, and also allows us to executed specific categories of test suites if desired at different frequencies during the build cycle.
• Criticality - if each test case in each suite is properly designed and bucketed correctly, a failing test case in the P2 bucket may not carry the same weight as a failing P1 test (assuming the failure is properly investigated). For example, failing BITS/BVT must be fixed immediately because they are considered to be breaking the build. Failing P1s should be fixed before the next scheduled merge into the main build, and failing P2s should be fixed before next milestone.
• Measurement - Occasionally managers will use test pass rates of certain types of test categories to establish exit criteria goals.
• Assignment - sometimes 'specialists' might be assigned to write performance tests, or approve performance tests. The same with fixing bugs in certain areas. Some developers are more adept at troubleshooting and fixing performance issues, or reviewing code structures for performant designs.

I am sure others can come up with other reasons to categorize test suites as well.