2

Should the tester check usability, security, reliability, anything else?

Improve this question
3
  • I think this question is too broad.
    – Bruce McLeod
    May 4, 2011 at 11:44
  • Agreed. Pavel, what are you hoping to learn from this?
    – testerab
    May 5, 2011 at 2:34
  • I hope to learn state of the art, common practices: what kinds of requirements testers should test. But now I see that this question is too general and argumentative :(
    – Pavel Surmenok
    May 5, 2011 at 4:58

3 Answers 3

Reset to default
5

Personally, I think yes, but no.

Yes in the sense that you need to make sure it's "good" in a common sense way. But once you've identified where it's not "good" you need to break down why into a functional requirement. Consider the categories:

Usability is not something that can be tested. How can you put into a specification "Must be easy to use" or something like that? enter image description here

Security is something that can be tested, but how do you know something is secure if you don't have a requirement that says what the definition of secure means? You could lock down every port on a server and call it secure, but it won't do anything.

Reliability is another thing that can't really be defined outside of the requirements. If it doesn't meet the requirements all the time, it doesn't meet the requirements. Period.

Something else that might fall in this category is efficiency. Someone might say "the app runs too slow." Well, how fast does it need to run? "Well I don't know, but this is too slow." Please. Come back when you have a requirement. Sure, I can take a look at the routines, and maybe come up with something, but I need to know what to shoot for.

Improve this answer
8
  • Thanks, glowcoder! I agree that most of these requirements are absolutely subjective, you can't say "ok, now this software is secure". But you can decide that certain things in your software have security vulnerabilities or that some dialogs are totally unusable and should be improved.
    – Pavel Surmenok
    May 4, 2011 at 5:25
  • 1
    I disagree - I include accessibility under usability. Regardless of requirements, keyboard shortcuts need to work, the app needs to work in high contrast mode, large font mode, etc. I also rarely see reliability requirments or perf requirements, but I test for them (e.g. what happens if I add and remove a user a thousand times?) For security, threat models, as well as penetration testing are outside the scope of requirements, but clearly test responsibilities.
    – Alan
    May 4, 2011 at 5:27
  • 2
    @Alan so while I agree that they should be tested and the results of that test evaluated, it should be translated and brought into a requirement. Otherwise, you end up with tribal knowledge of "Oh yeah, we also have to support large font mode." "Really? I didn't see that in my spec." "Well, we just do."
    – corsiKa ♦
    May 4, 2011 at 5:35
  • 2
    While I agree with above points, we also should keep in mind, what our mission or scope of testing is. If our stakeholders or our limits (time, money, people) doesn't allow for testing in certain areas or aspects (security, perf, etc,) we should ask them directly and clearly, if we should skip testing in these areas, do "some" testing or schedule it for later. In the end, they (often) shape our testing by setting the missions (e.g. "we need this for tomorrows exhibition"). Our job is to show them, what potentially could be a risk and make them think, if they are ready to risk it.
    – MaikNog
    May 4, 2011 at 9:56
  • 3
    @glowcoder - I see where you're coming from. Where we differ, I think, is our experience with complete requirements. IME, there's a lot of testing "between the lines" in requirements - i.e. I expect to do a significant amount of testing that isn't requirements-based.
    – Alan
    May 4, 2011 at 13:12
7

In my opinion, testers should absolutely be testing non-functional attributes (performance, usability, privacy, security, reliability, compatability, supportability, etc.). In general, I think testing should have less emphasis on functional requirements and more emphasis on non-functional requirements, but I don't want to side track your question.

If you're looking for a list, start with the ilities. If you're looking for a recommended list, it sort of depends on your product and customers. Overall, I find the list in my first paragraph a good place to start, but there's no one size fits all answer here.

Improve this answer
2
  • 1
    Thanks, Alan! Myabe I should clarify my question. I think that there are 2 points of view: 1. the tester should test only functional requirements (and the tester may be replaced by automated tests); 2. the tester should test everything, including appearance, color schemes, etc.
    – Pavel Surmenok
    May 4, 2011 at 5:20
  • In my opinion (based on my role and the test teams I work with) is that the test team tests for functional AND non-functional requirements.
    – Alan
    May 4, 2011 at 5:30
0

Alan's answer is spot on with the "-ilities". I also recommend that you buy a copy of Release It! by Michael Nygard - it's the key book for software developers and testers to help them understand what kind of things make software fail to operate well in Production. It will give you great insights into the kinds of "non-functional" or operational testing which you need to undertake on any non-trivial software system.

I identified several key chapters/pages in Release It! in terms of making software work well in production in a blog post here: http://blog.softwareoperability.com/2013/01/26/which-sections-of-release-it-help-to-make-software-operable/ These pages would be a good starting point for the kinds of operational testing you should be doing.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.