Testing without direction, how not to miss things? What about layout?

Question

So this might sound like a "complain" post but it really isn't meant to be. But often-times I will be handed a project, lets say a Website. And be told to give it a "Full run through".

So I take a look at it, take a look at any documentation (Usually just mockups if that, or any issue lists they had before) and go at it with every browser.

Inevitably i'll find some stuff, a good amount usually. But afterwards the client will usually have "design" issues. IE this needs to be padded 75x pixels to the right, or this image needs to be stretched etc.

Also, I am human...so there are things im going to miss. But just given the time (Aka a once over)...I guess I always feel bad about these kinds of things.

So my questions is:

1. Things are expected to slip through if you are just given a once through correct? Im not talking about a lot of major stuff. But little things (especially if it's a large website). How do I not beat myself up over this?

2. What can you do about design things? Is QA expected to notice these things? (Especially without reference). Like I will notice given a mockup for example that the heading should be blue instead of red, or the text should say this not that. But padding/etc... Im not really sure whats right or wrong....how do I amend this or improve this for myself?

Answer 1

Things are expected to slip through if you are just given a once through correct? Im not talking about a lot of major stuff. But little things (especially if it's a large website). How do I not beat myself up over this?

Yes. There are innumerable levels of quality and bugs will slip through at each of them. Consider these levels:

• Is it possible to use the website to do what the developer thinks it needed to do?
• Is it possible to use the website to do what the project manager asked for it to do?
• Is it possible to use the website to do what the project manager actually wanted it to do?
• Is it possible to use the website to do what the user needs for it to do?
• Is is possible to use the website to do what the user wants it to do?
• Is the website understandable?
• Is the website easy to use?
• Does the website actually help the business meet its goals?
• Does the business meet the owners' goals?

And so on. The longer you do this job, the more of these levels you will see, and the more problems you will see. You can never catch or solve every problem.

Beating yourself up because you do not achieve an unachievable goal is pointless. A more productive way to live is to pay attention to what you do, observe the outcomes honestly, and be receptive to change. This iterative approach can take you down a path of gradual improvement.

What can you do about design things? Is QA expected to notice these things? (Especially without reference). Like I will notice given a mockup for example that the heading should be blue instead of red, or the text should say this not that. But padding/etc... Im not really sure whats right or wrong....how do I amend this or improve this for myself?

Your organization has to decide whether QA is expected to notice these things. Sometimes a design thing is obviously wrong, but often you only know about a design problem after users have experienced it. You can test for that by engaging outsiders. See for example UserTesting.com.

Answer 2

Been in that situation, and far more than I'd like. While we'd like customers to provide complete and iron-clad requirements, it never (and I do mean never) happens. Well, unless you're shooting people into space. I hear they do a pretty thorough job.

For the rest of us, it's generally a continuum. The bits of this you don't have as far as requirements go, you'll have to rely on experience and make educated guesses. Obviously you know that exceptions, 500 errors, timeouts, and login fails are bad.

I've seen the "I can't test at all without requirements!!" mantra, and it's bupkis. You can test it. But it's not going to be "fully tested" in any sense of the word. :-)

I've found that sketching out my assumptions at the get-go helps direct my testing and gives me a good idea of how to prioritize/time box my efforts. This is especially important when "testing dark".

These are (always) some of the core questions I to ask. If the requirements answer them, great! If not, make your best guess.

Who is the tool/program trying to help? (target audiences) What is the tool trying to help said folks accomplish? This isn't merely what they "do" with the tool It's the difference between "getting a photo onto a website" and "sharing pictures with family" ... the difference is sometimes subtle, but very important.
Does it do that? (this covers mostly happy path) Reasonably easily? (this covers basic UI)

Once that's down, you (of course) will need more info as testing gets under way. I've found approaches like the following helpful in getting answers to the above...

"So is it more a experience assumed feature, or a level of experience?" <-- substitute examples as appropriate.

"I think this feature is supposed to work this way...is that right, or am I missing something?"

"Can you help clarify something for me, if I do X, should I see Y or Z happen? No? But Q should never happen, right? Ok. Well, it's doing Q .... "

I'd concur 100% with the 100% comments above. There is no such thing as a completely tested app.

You are not the "gatekeeper of quality" ... that's rubbish. Your job is to do your best to let those-who-make-decisions know the state-of-the-app. That job is bounded by time. Give me more time, I'll give you a more confident answer. Give me less time, I'll prioritize and do what I can but it'll be less deep.

This speaks to the ridiculous "complete run-through" request ... I generally will give a swag along the lines of:

In one day, I can hit 20% of the basic functionality, but only verifying to 5% deep. Give me a week, I can hit more like 40% of the most used functionality, but it'll be more like 10% deep. In a month, I can hit 75% of the functionality, with a 40% depth.

With the caveat: with more bugs found and longer fix times, those numbers will need to be adjusted.

Side note: I once had an old grizzled QA mgr ask me "How many QA engineers does it take to screw in a lightbulb?"

Uhh....

"None. We just report that it's dark."

True, but I'd add "but we will tell you when the light went out, how often it goes out, what the state of the switch was when it went out, whether we can make it go out repeatedly, in what room it went out, and who likely touched it last"

cheers - and good luck!

Answer 3

If you haven't been given requirements then it makes testing really difficult. You'll also be glad to hear that exhaustive testing is impossible (see The Principles of Software Testing)

In terms of design requirements, you're allowed to ask the product designer or business analyst for feedback/confirmation as you go because requirements often change - especially in an Agile environment.

It'd be prudent to put a few hours into test planning before any test execution commences. The test plan can then be approved by the client and they're effectively signing off what's in scope to test and, more importantly, what's out of scope.

My general rule of thumb is; if it's different to what you're expecting then raise a defect. This will prevent anyone in the business from going "Why didn't you say anything?" or "How did you miss that?" - but you'll have your back covered already with defects and a test plan which they signed off :)

Answer 4

1) Remember 100% testing is not possible, it is basic testing principle and everyone know that. so dont feel beat yourself too hard.

2) GUI bugs are important to the client so you should have concentrate on those, as they will be the 1st to be noticed. But px bugs can be done throug chrome F12 and checked.

Also a note: take your time, a QA should be given 40% of the time of development.

Answer 5

Q1. Things are expected to slip through if you are just given a once through correct? Im not talking about a lot of major stuff. But little things (especially if it's a large website). How do I not beat myself up over this?

A1. Accept that you can not be perfect. Make sure you have an outlet to note and record the bugs that you do find so they get addressed. Learn as much about the business domain as possible so your feedback relates to it. Make sure you cover different data, different devices, happy and sad paths, different connectivity, etc. and if you don't cover them in a "once through" make sure you have stated that. Remember you can have Safari, Chrome, Android Studio (and on a mac, iOS phone emulator and Parallels for other browsers such as IEs) all available locally or you can use remote services like http://browsershots.org/, http://browserstack.com or http://saucelabs.com

Q2. What can you do about design things? Is QA expected to notice these things?

A2. Yes, this is covered by exploratory testing and by using your brain to see if things look ok. This is also one of the reasons you have a lot of automated testing for forms and links as that frees you up to have time to manualy visually inspect the layout and try the look and feel to detect issues.

Answer 6

Things are expected to slip through if you are just given a once through correct? Im not talking about a lot of major stuff. But little things (especially if it's a large website). How do I not beat myself up over this?

I always say that I can test anything in any amount of time. When I'm given more time, I can do better. But I can still do good things when I have little time - just not as good.

Before you begin your testing, think it through. Since I have X hours to test this, what can I do that will have the most impact?

Perhaps you are testing on too many browsers, for example. When time is tight, I might skip the browsers that don't have many users in order to concentrate on the most-used browsers.

Or, I might not bother with any negative testing, until I'm confident that the happy path work well.

Prioritize the risks in your head first, make a few notes, then jump in and start testing until you run out of time. You'll be able to tell yourself that you gave it the best X hours of your time that your could. And you'll know that if things fall through the cracks, it was due to time, not ineffective testing.

What can you do about design things? Is QA expected to notice these things? (Especially without reference). Like I will notice given a mockup for example that the heading should be blue instead of red, or the text should say this not that. But padding/etc... Im not really sure whats right or wrong....how do I amend this or improve this for myself?

As always, it depends.

If the application you are testing is supposed to be consistent design-wise with other applications that you have already tested, then perhaps you will be expected to note any inconsistencies.

Certainly, you would be expected to note any internal inconsistencies within this particular application itself.

Sometimes, you can only do what you can do. And if the mockups don't tell you enough, you can't be expected to read minds.

This might help: http://www.allthingsquality.com/2010/04/there-are-always-requirements.html

Answer 7

Note before I dive in: We do Scrum so depending on the project methodology your company uses, this may or may not be useful for you.

Background:

We have our initial requirements from the stakeholders. The stakeholders are a combination of our clients, subject matter experts, product owners, executives, and managers. We are provided what they expect the final results should look like. This is after the project has been groomed by subject matter experts and product owners if nothing else. We have what is called a scrum team which comprises of the scrum master, our business analyst, our QA, and our technical associates. We get together as a team and review the project and the details they have provided from their grooming session. We make sure we understand what they are looking for to be the end product. Then we have a meeting to generate the BDD (Business Driven Development) which is loosely coming up with test scenarios of all possible things that need to be tested, including any regression test*ng to make sure we don't break existing functionality. Technical is part of this process. Doing the BDD makes a huge difference, makes more than the QA accountable for everything which needs to be tested and validated, and it makes the testing more thorough. And requirements do change. While testing you may discover a gap that the client hadn't mentioned. You bring it up. They then decide if they will absorb it into this project or it will basically become another phase of the project.

Sorry for the long reply. I just wanted you to know that true support of a QA is to include the whole team as part of making sure a project is tested as well as possible. I realize that may not be possible in your case. If it isn't, then you are limited to only one human doing the testing and validation.

As others have stated, there is no such thing as total, full, perfect testing. It's best efforts, sincere efforts, doing as much testing as possible. And as someone pointed out, a huge chunk of your time should be allotted to the actual testing. I've worked with QA's and know how challenging their jobs are (I'm a software engineer) and have respect for just how much weighs on them due to their testing is crucial to providing a quality product to the client. Do your best, don't beat yourself up, and with time it will get easier. This is honestly one of those careers where experience is your best asset. So know that it will get easier.