Tell me more ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.
up vote 2 down vote favorite

Should the tester check usability, security, reliability, anything else?

share|improve this question
I think this question is too broad. – Bruce McLeod♦ May 4 '11 at 11:44
Agreed. Pavel, what are you hoping to learn from this? – testerab♦ May 5 '11 at 2:34
I hope to learn state of the art, common practices: what kinds of requirements testers should test. But now I see that this question is too general and argumentative :( – Pavel Surmenok May 5 '11 at 4:58

2 Answers

up vote 3 down vote accepted

Personally, I think yes, but no.

Yes in the sense that you need to make sure it's "good" in a common sense way. But once you've identified where it's not "good" you need to break down why into a functional requirement. Consider the categories:

Usability is not something that can be tested. How can you put into a specification "Must be easy to use" or something like that? enter image description here

Security is something that can be tested, but how do you know something is secure if you don't have a requirement that says what the definition of secure means? You could lock down every port on a server and call it secure, but it won't do anything.

Reliability is another thing that can't really be defined outside of the requirements. If it doesn't meet the requirements all the time, it doesn't meet the requirements. Period.

Something else that might fall in this category is efficiency. Someone might say "the app runs too slow." Well, how fast does it need to run? "Well I don't know, but this is too slow." Please. Come back when you have a requirement. Sure, I can take a look at the routines, and maybe come up with something, but I need to know what to shoot for.

share|improve this answer
Thanks, glowcoder! I agree that most of these requirements are absolutely subjective, you can't say "ok, now this software is secure". But you can decide that certain things in your software have security vulnerabilities or that some dialogs are totally unusable and should be improved. – Pavel Surmenok May 4 '11 at 5:25
1  
I disagree - I include accessibility under usability. Regardless of requirements, keyboard shortcuts need to work, the app needs to work in high contrast mode, large font mode, etc. I also rarely see reliability requirments or perf requirements, but I test for them (e.g. what happens if I add and remove a user a thousand times?) For security, threat models, as well as penetration testing are outside the scope of requirements, but clearly test responsibilities. – Alan May 4 '11 at 5:27
1  
@Alan so while I agree that they should be tested and the results of that test evaluated, it should be translated and brought into a requirement. Otherwise, you end up with tribal knowledge of "Oh yeah, we also have to support large font mode." "Really? I didn't see that in my spec." "Well, we just do." – corsiKa♦ May 4 '11 at 5:35
1  
While I agree with above points, we also should keep in mind, what our mission or scope of testing is. If our stakeholders or our limits (time, money, people) doesn't allow for testing in certain areas or aspects (security, perf, etc,) we should ask them directly and clearly, if we should skip testing in these areas, do "some" testing or schedule it for later. In the end, they (often) shape our testing by setting the missions (e.g. "we need this for tomorrows exhibition"). Our job is to show them, what potentially could be a risk and make them think, if they are ready to risk it. – MaikNog May 4 '11 at 9:56
2  
@glowcoder - I see where you're coming from. Where we differ, I think, is our experience with complete requirements. IME, there's a lot of testing "between the lines" in requirements - i.e. I expect to do a significant amount of testing that isn't requirements-based. – Alan May 4 '11 at 13:12
show 3 more comments
up vote 6 down vote

In my opinion, testers should absolutely be testing non-functional attributes (performance, usability, privacy, security, reliability, compatability, supportability, etc.). In general, I think testing should have less emphasis on functional requirements and more emphasis on non-functional requirements, but I don't want to side track your question.

If you're looking for a list, start with the ilities. If you're looking for a recommended list, it sort of depends on your product and customers. Overall, I find the list in my first paragraph a good place to start, but there's no one size fits all answer here.

share|improve this answer
1  
Thanks, Alan! Myabe I should clarify my question. I think that there are 2 points of view: 1. the tester should test only functional requirements (and the tester may be replaced by automated tests); 2. the tester should test everything, including appearance, color schemes, etc. – Pavel Surmenok May 4 '11 at 5:20
In my opinion (based on my role and the test teams I work with) is that the test team tests for functional AND non-functional requirements. – Alan May 4 '11 at 5:30

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.