Should the tester check usability, security, reliability, anything else?
-
I think this question is too broad.– Bruce McLeodMay 4, 2011 at 11:44
-
Agreed. Pavel, what are you hoping to learn from this?– testerabMay 5, 2011 at 2:34
-
I hope to learn state of the art, common practices: what kinds of requirements testers should test. But now I see that this question is too general and argumentative :(– Pavel SurmenokMay 5, 2011 at 4:58
3 Answers
Personally, I think yes, but no.
Yes in the sense that you need to make sure it's "good" in a common sense way. But once you've identified where it's not "good" you need to break down why into a functional requirement. Consider the categories:
Usability is not something that can be tested. How can you put into a specification "Must be easy to use" or something like that?
Security is something that can be tested, but how do you know something is secure if you don't have a requirement that says what the definition of secure means? You could lock down every port on a server and call it secure, but it won't do anything.
Reliability is another thing that can't really be defined outside of the requirements. If it doesn't meet the requirements all the time, it doesn't meet the requirements. Period.
Something else that might fall in this category is efficiency. Someone might say "the app runs too slow." Well, how fast does it need to run? "Well I don't know, but this is too slow." Please. Come back when you have a requirement. Sure, I can take a look at the routines, and maybe come up with something, but I need to know what to shoot for.
-
Thanks, glowcoder! I agree that most of these requirements are absolutely subjective, you can't say "ok, now this software is secure". But you can decide that certain things in your software have security vulnerabilities or that some dialogs are totally unusable and should be improved. May 4, 2011 at 5:25
-
1I disagree - I include accessibility under usability. Regardless of requirements, keyboard shortcuts need to work, the app needs to work in high contrast mode, large font mode, etc. I also rarely see reliability requirments or perf requirements, but I test for them (e.g. what happens if I add and remove a user a thousand times?) For security, threat models, as well as penetration testing are outside the scope of requirements, but clearly test responsibilities.– AlanMay 4, 2011 at 5:27
-
2@Alan so while I agree that they should be tested and the results of that test evaluated, it should be translated and brought into a requirement. Otherwise, you end up with tribal knowledge of "Oh yeah, we also have to support large font mode." "Really? I didn't see that in my spec." "Well, we just do."– corsiKa ♦May 4, 2011 at 5:35
-
2While I agree with above points, we also should keep in mind, what our mission or scope of testing is. If our stakeholders or our limits (time, money, people) doesn't allow for testing in certain areas or aspects (security, perf, etc,) we should ask them directly and clearly, if we should skip testing in these areas, do "some" testing or schedule it for later. In the end, they (often) shape our testing by setting the missions (e.g. "we need this for tomorrows exhibition"). Our job is to show them, what potentially could be a risk and make them think, if they are ready to risk it.– MaikNogMay 4, 2011 at 9:56
-
3@glowcoder - I see where you're coming from. Where we differ, I think, is our experience with complete requirements. IME, there's a lot of testing "between the lines" in requirements - i.e. I expect to do a significant amount of testing that isn't requirements-based.– AlanMay 4, 2011 at 13:12
In my opinion, testers should absolutely be testing non-functional attributes (performance, usability, privacy, security, reliability, compatability, supportability, etc.). In general, I think testing should have less emphasis on functional requirements and more emphasis on non-functional requirements, but I don't want to side track your question.
If you're looking for a list, start with the ilities. If you're looking for a recommended list, it sort of depends on your product and customers. Overall, I find the list in my first paragraph a good place to start, but there's no one size fits all answer here.
-
1Thanks, Alan! Myabe I should clarify my question. I think that there are 2 points of view: 1. the tester should test only functional requirements (and the tester may be replaced by automated tests); 2. the tester should test everything, including appearance, color schemes, etc. May 4, 2011 at 5:20
-
In my opinion (based on my role and the test teams I work with) is that the test team tests for functional AND non-functional requirements.– AlanMay 4, 2011 at 5:30
Alan's answer is spot on with the "-ilities". I also recommend that you buy a copy of Release It! by Michael Nygard - it's the key book for software developers and testers to help them understand what kind of things make software fail to operate well in Production. It will give you great insights into the kinds of "non-functional" or operational testing which you need to undertake on any non-trivial software system.
I identified several key chapters/pages in Release It! in terms of making software work well in production in a blog post here: http://blog.softwareoperability.com/2013/01/26/which-sections-of-release-it-help-to-make-software-operable/ These pages would be a good starting point for the kinds of operational testing you should be doing.