What advice can you give for encouraging Project managers, and Developers to factor in tests for purposes of proving an application system's performance, resilience, capacity, scaleability etc beyond the testing conducted at unit testing stage. There is more emphasis/reliance on citing the efficacy of the design concept (and Unit testing) as the sole insurance for meeting such requirement. In short, also assuming that the system will run as designed under the less controlled circumstances i.e PROD like it did in the unit tests. Consequently, we may not see monitoring / evaluations of performance results from the functional tests conducted in the UAT period. We need to use UAT gauge the underlying system performance to see if there are performance limitations under more production-like circumstances when all the processing chains are integrated for the first time. However, we are finding that the project solely focuses on the Business results. See also article Why is separating functional and non-functional test important (or not)?
-
I am having trouble relating the first sentence to the last one. Perhaps you use some of these terms in a different way than I do. Can you elaborate on what focus tests have to do with capacity and scalability?– user246Commented Apr 2, 2012 at 17:13
-
I'd almost answer this with Load and Stress testing, since this seems to be what is being asked in the first sentence. But that would not mean less focus on testing during the UAT phase if you are looking to front-load testing...which is how I would read this.– MichaelFCommented Apr 2, 2012 at 17:55
-
Hi to clarify the problem we face is that e.g. load tests are done only @ the code unit testing stage. Whereas, it would be good to have results from the UAT or integration test Phase demonstrating that the system performes well, no data is lost on transit from source system to target system when the system is stressed. Sometimes the test evidence given for the A --b delivery is only supplied using a small sample from the unit test phase, and not from when the systen is tasked with processing a day's worth of data, and all the variables during UAT.– Pernilla OCommented Apr 3, 2012 at 10:32
-
1Not worthy of an answer, but I'll plug a paper I wrote last year that may help. uploads.pnsqc.org/2011/papers/T-30_Page_paper.pdf– AlanCommented Apr 5, 2012 at 21:27
-
Thanks Alan. Useful especially the point about complementing functional verification with tests designed to probe the area of customer experience and gauging the customer's perception of quality. In our project, we will have to get Business users to articulate their expectations in terms that go beyond the strictly functional - they should examine and articulate their requirements for availability and system performance in practical terms. We will need to encourage the PM's etc to explore this territory with acceptance criteria definition etc. Snr Executive support will increase % success– Pernilla OCommented Apr 10, 2012 at 10:37
7 Answers
Suggest having dedicated Non-Functional (NF) Test Management practice separate from the functional practice (UAT etc.). Also NF needs to be much more closely aligned to (i) Infrastructure Architecture, and (ii) Production Systems Management. Also have the NF testing as separate test phases with separate governance. Have used this approach successfully with a number of clients and it works much better than having combined Functional and Non-Functional. Be prepared for Functional Test Management to want to govern NF - they will always fail to understand it and miss-govern. The separation really needs to be at the CIO level.
First, you're going to have to understand the Project Managers and Developers that you work with. Not knowing them it is hard to understand what sort of arguments might persuade them. Think back on arguments you've had with them, what kinds of arguments did they use to try to persuade you. In all likelihood a similar form of argument will likely hold sway with them. For instance, if they are always pointing at customer data, you need to form your argument using customer data in some way. Talk to them in their own voice.
It's also possible you can also appeal to company culture in some way. If your company prides itself on being frugal, thorough, well-designed or user-friendly those are the terms to use when forming your argument.
Finally, I think you need to clarify in your own mind what you're trying to accomplish. Don't test in a vacuum. How is better performance testing, for example, going to further your companies business goals? Can you explain your reasons to yourself or to a teammate in a clear single sentence? Spend some time tearing your own ideas apart, make sure they're defensible.
-
sqa.stackexchange.com/questions/2947/… Wise words Sean. Some problems may go undetected until PROD release. Non technical colleagues may not appreciate how things can unwind & won't see the merits of tracking processing, execution times, exception etc as long as the data is delivered & they get their reporting results. Commented Apr 3, 2012 at 11:05
-
sqa.stackexchange.com/questions/2947/… It's only when the problems show up in PROD that the point is understood more. Developers tend to test delivery from a -b @unit stage, But this doesn;t cover all realistic amount of data and all the variables of when the system is stressed. More attention to the system aspects of test results in UAT would help Commented Apr 3, 2012 at 11:11
-
To help others realise the value of your testing why not have your test environments match your production environment as closely as possible and invite "users" to come over and have a play with the system & new features?– devonpsCommented Apr 3, 2012 at 15:03
You need to connect the dots. There are at least three messages to convey: your production system has problems that are worth worrying about; those problems result from your shortcomings in your organization's current practices; and the best way to address those shortcomings is to introduce new tests, i.e. to use your terminology, additional UAT-level tests focusing on the application system's performance, resilience, capacity, and scalability.
You did not describe the nature of your production problems. You also did not say whether the project manager and developers are aware of those problems or share your concerns about them. Your first message may need evidence about the frequency/severity/impact of those problems. Specifics are more compelling than a general sense of unease.
Sometimes production problems are caused by factors downstream from the development team. Your second message needs evidence that these problems could be traced back to the development team and not, for example, to sloppy practices in the production team or infrastructure problems at your ISP.
Finally, your need a persuasive argument that investing time in new UAT-level tests is more important than other things, because there are always other things. Adding tests may slow your delivery time, require spending more money on headcount and/or hardware, and/or delay the introduction of important new features.
-
@user248 Spot on, One of the drivers behind this qu is to uncover unknowns inc processing bottleneck/infra/downstream isues prior to Prod release & prove that they don't exist.Or do exist so we can fix them Commented Apr 5, 2012 at 16:39
I lost you somewhere along the question... Anyway, upper management are usually convinced by showing them the ROI, for example if you can show problems that could have been detected and solved during a non functional testing phase.
-
sqa.stackexchange.com/questions/2947/… True. It would be good to have this testing covered up front tho' without waiting for the drama to unfold first ... Commented Apr 3, 2012 at 11:31
At my org, we have 'bug bashes'. Besides structured testing, you invite others who are not very familiar with the feature to have a go at it with the sole intention of destroying the software. Anything and everything is fair game. We've found that this uncovers the most number of bugs. :)
Project managers ultimately report to a more senior person about how the project is going. By performing more types of testing, you can give the PM more ammo to provide to the senior mgmt about how reliable, scalable the system is and - assuming this is a problem across the company - how innovative the project is.
To do this you will need to be able to quantify the effect of the extra testing you are doing, otherwise it will be seen as wasted effort. Give the PM a graph of how the speed of the system has improved over time. The PM can then show that to senior management and you both look good.
For me, the terminology "non-functional" is part of the problem, as it immediately implies a lack of something. Some people use "cross-functional" instead, or terms like "constraints".
What I have found works well is to talk in terms of Features: end-user features and operational features. Focusing on the operational aspects opens up a dialogue with the Operations team to cover things like performance, scalability, resilience, etc., and talking about both groups of requirements as "features" means they tend to get treated as the "same kind of thing" and can be prioritised together in the same backlog.
Finally, by having the product owner responsible for the operability of the software (i.e. she gets woken up at 2am when the site goes down), you soon see "non-functional" or operational concerns prioritised higher in the product backlog!
A recent article in the DevOpsFriday newsletter covers this (note: I am the interviewee in the article): http://blog.softwareoperability.com/2013/05/05/software-operability-article-in-devopsfriday/
Hope this is of use.
