Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It only takes a minute to sign up.
Sign up to join this community
How do quality assurance engineers test calculations done inside the code and calculations in stored procedures?
Also if the calculation is complex what should be the approach to test something like that?
We can't rely on developers codes for complex calculations and even if a QA developed a method to calculate it, how can we rely on it? Because even the QA could have made mistakes.
Im expecting an answer which is suitable for both manual and automation QA's
All engineers (application and automation) test algorithms by providing known inputs and having knowledge of the expected output in order to perform verifications.
The verification values can be found by manual calculations and/or different algorithm(s).
If the algorithm and manual calculation approach are not known another approach is BDD (Behaviour-Driven Development). What formula would work for "1". Then "2". Then "3", etc. This does assumes that you can at least manually calculate those. At some point you make the call that it seems reliable enough with enough numbers tested. This is an approach to use tests to discover the design. If you can't figure out what your formula should produce given a simple input, e.g. "1", you may have a different issue.
We can't rely on developers codes for complex calculations and even if a QA developed a method to calculate it, how can we rely on it? Because even the QA could have made mistakes.
Have you tried test driven development? One calculation behavior at a time? A calculation consists of mutliple steps. For example (1 + 2) / 3 = 1 consist of two steps, which you can test separately. E.g. breakdown the calculation and test in smaller parts. Make a couple of end-to-end tests.
It is harder to make mistakes when you work in smaller steps.
Your question explicitly assumes that QA must be 100% infallible in order to be useful. This is incorrect, and I'm afraid shows that you have seriously misunderstood the purpose of QA.
Because even the QA could have made mistakes.
Of course they could. Good practise is to consider reviewing and testing as equally prone to errors. As a result, you do not blindly make changes proposed by a reviewer, and you do not blindly make changes to make your tests pass, because both could be wrong.
The purpose of QA is to provide defence in depth. The more eyes on a problem, the better it'll be tested. The original designer is generally too invested in the design to see outside the box and will tend to test against what they know it does, whereas an independent engineer can approach it as a black box and test against the requirements without preconceptions. This can also expose issues with ambiguous requirements, where both the designer and tester can be "right" for their interpretation, but still come up with conflicting results.
It is not possible for QA to guarantee that there are no bugs in the code. Be clear that it is not just impractical, it is actually mathematically impossible. (Read up on the halting problem if you want proof.) What QA can do is run a reasonable set of tests which will exercise the code over a reasonable range of operating conditions. The more detailed the tests are, the more likely they are to find any remaining bugs, but also the more they cost. Development standards such as DO-178B often mandate the level of testing required depending on the safety-related nature of software.
In the 1980s and 1990s there was a significant drive from academia to use formal specification, in the belief that the process of turning designs into code was a significant source of errors. A formal specification language/process was intended to allow test cases to be generated which would be unambiguously correct, and hence could be used to validate the possibly-faulty code in the way you seem to want. As you can probably see (but apparently Computer Science "experts" at the time couldn't), the formal specification therefore became a second implementation of the requirements and as such was equally prone to contain errors in its implementation. Industry discovered that in fact the process of coding was relatively robust, and normal QA processes were substantially likely to pick up coding bugs. Most serious problems turned out instead to be related to incorrect or ambiguous requirements, which inherently cannot be solved through any technical process. Most people involved in the formal specification field at that time acknowledge in hindsight that it was a dead end.
And as a step beyond QA... Some safety-related systems use redundancy where two (or more) control systems are produced by independent non-communicating teams. All steps from design to QA are carried out separately for each system. Whilst both systems may still contain bugs, they are extremely unlikely to both contain the same bug, so a situation which causes one controller to misbehave will not affect the other controller(s). Whether the fault is due to a coding bug or a different interpretation of requirements, one controller will still be operating correctly, and the system as a whole is designed to ensure safe operation in the event of this happening. This illustrates another principle of safety engineering - assume that the fault will occur, and figure out how to make the system as a whole robust to that fault wherever reasonably possible.
Each situation is different, but there are a few things you could test for:
Correct formula First thing to check for is if the specification actually does the correct calculations. In science checking for units is one example: dividing meters by hours will not give meters per second. Peer-review is a good way to do these checks.
Correct answer One way is to create a table in a different program with inputs -> outputs, say using excel, and then compare against these values. Use both typical input values and untypical. It is a good idea to look at both the algorithm as such and the implementation to find, what I tend to call, pivotal points and test extra around these. Division very close to zero is one example. Automate this test as much as possible.
Inputs at or out of bounds In computers, algorithms always make assumptions about acceptable range for input values. You might send in a integer, but might not accept negative values (or very large values, ... ). Values at and outside the bounds (say maximum integer value) should always be tested. In my experience, if there is no explicit description of the allowed bounds, the calculations will most probably fail eventually. The calculation should behave in a "designed" way when values are out of bounds.
Desired precision In algorithms or calculations you can easily drop precision unless beeing very careful. Maybe you do the calculation using single precision floats but it really needs double (or more) in order to give correct answers. Sometimes this can be guessed, say when a very small value is added to a very large value or when you divide by a value close to bero.
Run time Some algorithms are supposed to converge to a value, say Newton-Raphston. But is this absolutely sure? Are there situations when the algorithm does not converge, or converge extremely slowly? This should be verified and the implementation should have some guarantee on run time behaviour.
Quality Assurance often takes the form of trying to do the same thing as the function under test, but in a different way. Simple sanity checks in hardwired or easy-to-trace cases often catch errors. In a way it is the same thing as confirming an experimental result: More samples, differently biased, that tell the same story corroborate the conclusion and build confidence in it.
Take several fiberglass mesh window screens, each with several larger holes, and place them over each other. Taken over random orientations, and given enough screens, chances are good that there will be few or no holes or gaps consistently. Think of it as a sieve like Fermat's probabilistic primality test. Error drops off exponentially with each new orthogonal take on the question. In reality, our biases tend to align more than they are disparate, and so the return on investment isn't as high as the theoretical maximum.
An example of automation might be to take an optimized implementation of an algorithm and spot-test it against samples generated by a naive implementation that solves the same problem. The naive implementation, though inefficient, is almost always simpler and so can be thought of as more reliable than the universe of optimized implementations.
Due to human error, no mortal's QA initiative is guaranteed to be perfect, with or without automation.
2+2 = 4.
"Keep it stupid simple".
No matter what method one may use or how complex is the calculation, simplest & most effective approach IMHO would be reading inputs and corresponding outputs from external file as hard coded values.
Any complex calculation can be considered mathematically,simply a function of mapping of an set of input values to an output value.
I have been in exactly same situation.Calculating those complex values was not an challenge once business manually prepared and shared an excel with all primary complex scenarios to entire Dev/qa team.The real problem is calculating them in place in test code which may change and break over time.
Of course you should do some basic checks, but often there are much more powerful tests that can reveal an error somewhere, even if they don't pinpoint its exact location.
For example, iterative algorithms should converge at the correct rate as predicted by the math - "converging faster than expected" is just as wrong as "too slow", unless you can explain why.
In my own field of solid mechanics and dynamics, there are many "self-checking" tests that can be done because the computer model is necessarily constructed using some definite coordinate system, but the physical object being modelled "knows" nothing about that. Therefore, if the results depend on the coordinate system being used, they are guaranteed to be wrong.
Tests such as that can be very powerful, since you can create a completely "arbitrary" set of test data where it would be impossible to check the results by hand, or by comparison with published data, and then "randomly" (and automatically) transform that input data set in many different ways, all of which should give the same results.
Of course that type of testing doesn't necessarily catch gross errors (or malevolant programmers) creating a function which always returns the same value whatever its input, etc - but code reviews, not testing, there to stop that sort of thing.
Another important design principle is to use algorithms that can be tested! That applies at all levels of the system design and implementation. Good engineers do that when designing anything, not just computer software.
Those are just a few of examples that non-specialists should be able to understand - similar testing methods for mathematical algorithms can go much deeper than those relatively trivial examples.
For something like a stored procedure I would use a contract testing approach , this would be intelligently mutating set of input and output pairs. You need to focus on testing the specific effects of the procedure and not on how SQL quirks are implimented.
For a something like a function with massively variable parameters you should use property testing, this is where you attempt to run all meaningful combinations of inputs and observe certain properties either as constant s or as sequenced values. E.g. proving acceptable uniqueness or entropy.
A good test plan should include a variety of approaches that are designed to layer up confidence in the software when their results are combined.
It is not a problem to use the algorithm/formula to calculate the results, this is known as the oracle - possibly with a different known implementation. You have to choose the inputs and outputs right by :
equivalence class test - sets of input that have the same output
boundary value analysis - the above groups with values chosen where the output changes
