10

I work as a QA/test engineer in a small software company. It is a web based product. I was assigned a few projects by my manager. I mostly do functional/black-box testing. Before the release, I tested many good scenarios and found quite a few defects in the code. Most of them got fixed. I did my analysis and gave important information to the stakeholders. I had a good test strategy at that time. Now, a client found a lot of bugs and the client is disappointed with testing.

So how to improve my testing and cover all test cases?

My manager and the stakeholders don't trust my work as a tester, as how dare I say that it PASS the QA the first time. They think that I'm careless, not worthy of working, and risky.

  • 1
    The defects the client found, are they obvious? Do you also think you could have done a better job? Or is it just new insights? They think you are careless, but what do you think? Could you add your perspective after the fact? – Niels van Reijmersdal Jan 15 at 9:23
  • 2
    Don't forget: the developers put the bugs there. They should share any blame. – Pete Becker Jan 15 at 13:56
  • 8
    @NielsvanReijmersdal this is blatantly plagiarised from the linked question. I can’t see how it’s not an exact duplicate. – Tim Jan 15 at 14:35
  • 2
    @Tim Wow, yeah didnt see that, seems to be just copied. I flagged it for moderation :) – Niels van Reijmersdal Jan 15 at 14:41
  • 2
    Anyone that thinks that having a tester means that there will be no bugs is an absolute idiot. – Reuben Mallaby Jan 15 at 18:53
16

You will never ever cover all of the possible test cases.

and found quite a few defects in the code. Most of them got fixed.

Were any of the bugs related to these. Did you point out the possible consequences of the ones that were not fixed?

I did my analysis and gave important information to the stakeholders.

Did anyone read it? Did you point out what you did not cover?

Now, client found lot of bug and client disappoint with testing.

Were they in the same area? Was it anything that you could have changed? When told about it, did you realize other things that you could change about how you test?

You were assigned a few small products. I'm guessing that a few bugs were probably found in production, because, well, they always are. The important part is to learn from it. Don't beat yourself up over what you've missed, just learn from it; don't miss the same thing again. Keep doing a good job; build up your reputation in the eyes of your manager and stakeholders again. It may take time. They may not realize the nuances of testing either. It's not like they treat the developer the same way for writing the bug to begin with, are they?

You say that there are managers and stakeholders. Is the client another company? Do they do User Acceptance Testing. Could they take a look at it before it goes live? You're one person, you, or an entire team, will ever catch every possible bug. Just catch the ones you can, learn, and move on.

10

So I can also give you an example in our situation. We had not exactly the same issue, but at production we also found a lot of issues and we were facing some troubles in the past. There were some scenarios how we improved the quality

1. All bugs found, create test cases for it (at least for the cricital ones): For bugs which we founded we created test cases for it. Based on the Release Policy with bugs marked as "high" "very high" or "critical" we created test cases. But we also broadended the test cases with founded bugs. An example would be this; For example the shopping card unfortunately didn't allowed to accept payment via credit card with Master Card (this was a bug). In this case we created test cases with Master Card, but also with other payment possibilities (Visa, debit, payment in advance etc.). And furthermore we also checked the VAT (value added tax) and also broadened the test case somehow in an E2E test case (e.g. payment was done with web based application on browser firefox, user checks again shopping card in his mobile app version with safari). Afterwards you can automate these scenarios (whenever possible) and broaden your test scenarios. We did this and based on a test report we showed this also our Product Owner.

2. Document exploratory testing with tool: For us it was also important to document all our test cases. Our tool - which we used was HP ALM. But we wanted to record the scenarios so hence we used tricentis. Tricentis was able to capture all the steps. This was very helpful because we had also an issue with a country which was available temporarily (there was a selection box 30 countries had to be displayed but one country was mising temporarily - this was really strange). With this tool we captured all our test scenarios it even created test cases. This was a good help, because this saved time and was somehow a "evidence tool" (in case that the country was not available again).

3. Involve users in your test scope (or business department): Try to invite also real users in your test scenarios. This could also be the business department. Because sometimes the testing team just is keen on executing test cases which has been defined ("testing blindness"). But the user or the business department does a quite different job and know different procedures or steps to use the application. For example as tester we entered values manually in a text field, the business department is lazy and just copied text values in fields and within this application we got a throwed exception. We would never find the bug/defect without involving the business department.

4. Speak with developers: Sometimes the developers can give you good examples how to test the application. I would also discuss with them especially about boundaries. For example in a text field only char values are allowed. Then you should ask yourself as tester what happens when you enter an int value or just special characters (e.g. @ & %). There are a lot of tricks and things to do for testing. But some problems can be solved with the development guys in advance!

5. Discuss with the other stakeholders and elaborate a test strategy: In our case we had two testing departments. So the IT department was mainly just testing the uster stories and hence focussing on user stories. The business department on the other hand was doing the same thing (!). Just retesting the same user stories again. Well, could be useful, but at the other hand we didn't considered e.g. layout issues, language issues, E2E functionalities were not working...Why? Because we hadn't defined it in our test strategy / test concept! So maybe you should check again your test concept and talk with the other stakeholders about your test strategy

6. Product available at market? Listen to your users: In your case I am not quite sure what application you are testing. In our case it was also a app which was available at the market. And with Apple Store for example you can check the users comments. Some users complained and were very unsatisfied about the product and some of them even complained about known errors. So we considered also these comments in our next test session. Sometimes users perspective give a really good feedback about what we did not covered for our testing scope. Furthermore you also can find good suggestions for your product owner :-)

So there are just a few hints, but in our case it helped to improve the quality - of course there are a lot of more things to do. You can never stop with testing and testing never gets bored :-)

But could you please explain or give more about your application which you are testing. Maybe I can give you more hints.

7

The biggest problem with failure is not learning from it

Management should will be grateful if you

Acknowledge the failures and put together a plan to address them and future occurances


Did they say all those specific words:

  • You are careless?
  • You are not worthy of asking?
  • You are risky?

Or are those words you are using, based on your - very valid - feelings about being treated so poorly? Did they perhaps say something more along the lines of

  • How come you missed these?
  • Why didn't you detect these before the client?

Which is, lets face it, are reasonable questions! However they might have been rude and nasty, phrasing it as 'what the hell, how could you possibly miss this' which will give you those very human and very normal feelings.

Work through the emotions and get to where you are focusing on the product quality.
Talk to management about a plan to:

  • measure quality based on customer usage
  • Get more feedback from customers faster
  • learn from feedback about what breaks a lot
  • do more testing
  • do more appropriate testing
  • review unit tests as well as UI tests

Don't leave the situation just with 'intentions to be more thorough next time'. Without a concrete plans to address identified issues, they will reappear. If they do, next time you will be in trouble. Mistakes may be accepted first time. Repeat them and you may be fired.

  • I hope there would be a solution from stack overflow team itself for this. Just down voting without any explanation or helpful comment – PDHide Jan 15 at 11:38
  • 1
    If PDHide's comment seems confusing, it is due to my comment about downvoting being deleted. So now, along with downvotes, we get delete of comments about it? Becuase they should have been on meta perhaps? this gets sillier and sillier over time and the community does not benefit. Worst is when I see about 6 different answers all downvotes at the same time with no comment. That just seems silly and insensitive to fellow members. – Michael Durrant Jan 15 at 13:41
  • 1
    SO has minimum requirements for length of text and changes etc in various areas, a requirement of a comment of at least x characters is one idea here. Of course this would also be revealing identity (unless hidden) which, unfortunately would probably be a very bad thing and lead to all sorts of silly, personal, retaliatory behaviour. So hide the person name would prob be needed. This is a complex problem. – Michael Durrant Jan 15 at 13:44
4

This might be a tricky situation, especially if you work with people who have never worked with testers, or have not much experience in software developement in general.

I think there are a few important points you and your colleagues need to realise:

  • You will never cover all the possible test cases. That's not even not job, because then, you'd be extremelly ineffective, focusing on unimportant things. Find out what's most risky and test primarily that.
  • You are not QA, you don't decide about people on your team, you have no say in the budget, you don't decide when the deadline is. But all of these affect quality. You are a tester who tries his best in a given situation. If you have no time to cover some important test cases, you should communicate this very problem to the team and let the project manager decide (and be responsible for the outcome).
  • The whole team should be responsible for the final outcome. If you, or anyone else, feel you're the last one who can uncover all the bugs hiding in the system, you are wrong. You'll miss things, you'll forget to test something. The sooner you and the rest of your team realise this, the sooner you can find ways how you as a team can minimilise such situations when a bug slips into production. It might mean that developers start with unit testing, they might introduce code review, you might get a new colleague, more time for testing etc. Whatever the solution, it will likely be a team effort, not an individual one.
  • However, the previous point should not mislead you into thinking you can't do more. There're ways to improve. You mentioned that client found some other bugs. Wonderful, it seems the client knows what they want from the system. Find ways to talk to the client before the system gets released into production. Test together. Let them do UAT testing. Let them use your test environment for a day before the final go/no-go decision.

There might be more to it, these are just some points that come to my mind right now. The takeaway is quality is a team effort, not an individual one, no matter how confident you feel about your testing skills. Find ways how other around you can do more testing, and you all will be more successful in finding bugs.

  • OP says "I work as a QA/test engineer in a small software company.". You say "You are not QA" ? – Lightness Races with Monica Jan 15 at 11:50
  • @LightnessRaceswithMonica: See the title. – pavelsaman Jan 15 at 12:14
  • I can see it. What about it? – Lightness Races with Monica Jan 15 at 12:15
  • 1
    A software tester is somebody who tests software. That's QA.... – Lightness Races with Monica Jan 15 at 13:01
  • 2
    It sounds like you are confusing "QA" with "Manager of the QA team". Testing software and hiring people are two completely different things – Lightness Races with Monica Jan 15 at 13:10
3

All other comments have explained the situation and solutions pretty comprehensively, but i just want to add few thoughts:

and found quite a few defects in the code. Most of them got fixed.

Here you have to ask yourself some questions,

  1. Where these bugs critical in nature ?
  2. Does it require hard effort to identify these bugs?

As discussed in many threads, defect number is not a KPI to measure efficiency of the testing. Its great that as a tester you have came across many defects

But you should think that , does this mask the areas that really need the check ?.

There is something called "absence of errors fallacy" which means that the bug fixes does not matter if the product it self is unstable or not worthy of using.

What you can do?

This situation can be or could not be your mistake, you should validated whether

1. You had enough time to test the product?

If not you should clearly mention it to the management before approving the build as QA pass. You should insist on having more time and not releasing the product unless you get enough time to achieve a fair test coverage.

If more time is not feasible , you should clearly document it or communicate it with stakeholder before releasing

You should also try to do smoke test and sanity test , if you don't have enough time to do complete regression.

Try to automate regression tests , this will reduce your work in next sprint . You will get more time to concentrate on new features

2. Did you had enough support?

If you are the only QA and the test task is more than you could handle, then communicate it properly and request for more resources.

3. Did you had proper acceptance criteria and documentations?

Don't take blame on your self, what ever the issues communicate properly. If the user stories lacked proper acceptance criteria then discuss in retrospective and clearly state what you where lacking. Ask for proper product documentation

4. Know your self worth

The role of QA is close to business and hence can be powerful. Don't hesitate to ask for more details, support ,discuss issues, etc. A strong QA is not a person who finishes task in time, but who takes the ownership of quality software process.

What ever you do document it , be ready to say 'NO' when you have to. Ask for clarity when it requires, and suggest improvement in the entire SDLC if you have any. Don't pull yourself back

Summary:

  1. Properly communicate requirement
  2. Ask for support when needed
  3. Be ready to say NO to testing when there is not enough time or documentation
  4. Learn from mistakes
  5. Automate regression tests and concentrate more on critical areas
  6. Always do smoke and sanity test before even starting proper test
  7. Do regression and cross browser test before releasing product
  8. Don't be afraid to ask and say NO
  9. Don't be afraid to reject a build if its not stable enough
3

However exhaustive testing is not possible. we as a team consider following facts which might affect the coverage of Testing:

  • Effective Communication with Business Analysts: I believe testers should have proper understanding of what to test. Testers should be explained what problem clients are facing and what could be all possible solutions and why this solution is chosen. Testers should read the BRD thoroughly. Testers should have domain knowledge same as BA. Testers should have clear conversation regarding their doubts with BA team. They can also provide their inputs at this stage regarding Scope miss. So that leakage from requirement gathering stage can be as less as possible.
    Despite of this, testers also need to read FRD, FSD and should be aware of how we as a team providing functional solution for the requirement and how feasible that solution is.
  • Communication with Developers: Instead of only being a gatekeeper, tester can provide inputs in development part and developers should also have to play a role to achieve quality of Product. As responsibility of Quality is not only at a QA but it is whole team's responsibility. Testing team can implement "Preventing Bugs" model OVER "Finding Bugs" model. For example for web application testing we as a testing team built some common component checklists as well as common validations checklists which will be referred by all developers at the time of development and there is less chance of missing those things in any of the build. When any functionality compromised and team needs to stand for it, it's both tester and developer's responsibility to take ownership of their work instead of playing blame games. Communication for bugs, steps to replicate each bug should be clear between developer and tester. If any bug is making too much hurdle and it can't be reproduced, tester should have to communicate that bug with developer because developers are the person who know what is written there in code and which part of code is causing that defect.
  • Understanding what all can be missed from development perspective: As explained in the above point, developers most of the time knows which part of their code can throw exceptions in some conditions. So, it is good to have quality of a tester if he/she can judge where system is going to fail. API Testing should be performed by Testing team. Testers should have experience/knowledge of the platform on which product is built.
  • Understanding Database architecture of the system: It is said that Performance tester should know what's "under the hood". Data is an important corporate asset. If current approaches are not sufficient and testers can test database and can provide their inputs then it is too good for any product. Testing provides the concrete feedback required to identify defects at an early stage. Testing database also support for evolutionary development.
  • Communication with Product Owner: It is important that testing team keep updating the Product owner about scope of their testing, no. of defects, Which are highly critical, which are of medium priority, and which are low priority as well as what would be impact if we provide release without resolving the defect. Release notes would not only be written but release notes should also be explained to product owner and product owner should be aware of what is the status of quality we are providing in this release.
  • Communication with Peer Testers: Sometimes miss of a bug is because of ambiguous communication in between the testing team. Team objective should be clear and everyone should have clear understanding and should work on same motive. Each team member should discuss the difficulties faced by them in their daily work and what solution he can figure out and what solution as a team can add into process. If one of the team member is on leave then or coming late and if it is causing hurdle then team have strategies defined and clear for responsibility transfer and ownership of transferred responsibility.
  • Writing Use Cases use cases, test scenarios, test cases, & their mapping: This is the essentials for any QA. But will vary based on requirements of different projects. But I suggest to write down Use cases first then writing Test Scenarios. Mapping use cases with requirements. If requirement do not exist for a use case then consult with BA or Product owner and get that requirement added in scope and then mapping requirements with scenarios. Mapping of scenarios with Test cases. This would create the picture so clear and it makes the chances of mistakes less. As even if you miss a requirement, you can caught it. If you missed any functionality check, you can caught it in scenarios review. If you missed any specific Test case check or checking any specific pre-condition, it can be caught easily.
  • Execution & tracking: Tester is owner of defects in the system. It is part of tester's job to get the defect logged as a bug against test case. If test case is missing then adding test case. Capturing and adding all the related test cases if found any test case. Getting bugs resolved from developers on time. Taking decisions regarding which defect team can afford at the time and when exactly it should be resolved.
  • Learning from your own past mistakes as well as your peers' mistakes: It is always required in any profile you work. Even it is required in your personal life.
  • Review from peers & leads:
  • Review of own task on next day or day after tomorrow.(If affordable):
1

A few things can be considered here:

  • Creating some kind of coverage matrix or document in which you identify what you will and will not be covering in your test cases. This can then be agreed on with you customer ahead of testing. This could also prevent/safeguard you from missing any (big) functionalities as you customer would highlight anything they expected you to include in this.

  • Taking screenshots or video evidence of your testing. This way you can prove you actually did all your tests and that they were OK at the time of testing

  • Retesting after the defects have been fixed. You mentioned you found quite a few issues which were then fixed. These fixes could have introduced new issues in functionality that was previously fine. No need to retest all of your test cases but you should retest related functionality, functionality that has failed before, high risk/importance functionality, etc.

If you do this and have this documented or reported somewhere, you should be able to (re)gain some confidence from you client in your testing work and value.

0

1) "testing" is a subjective term ... video game companies with QA dept's test their games to see if they work within specified parameters. Meanwhile a dedicated gamer with way too much time on their hands will always find some exploit or cheat to use in a speedrun or what-not. This person is just spending way more time to dig into the system, time that would not be cost-beneficial to the company to QA. And, sometimes they're (ab)using the system in a way not intended. (EG: a game that's intended to be played and enjoyed suddenly turned into a speed run that it wasn't intended for).

So, you have to first get the client to explain to you the bugs and issues they found. If they found really, really obscure stuff that wouldn't come up unless someone was really looking for trouble (IE: outside normal use-case scenarios), then they're idea of "testing" and your idea of testing on not on the same page (not apples to oranges).

2) "Testing" should be done in Taguchi style fashion, IE: you have ideal parameters the system should perform under, but also you should test it for semi-obscure ways it could be used that you could forsee someone (ab)using it with.

Taguchi was a Japanese statistician and process optimizer. He advocated for machinery to get built to operate within very broad spectrum of (ab)use case scenarios, then fine-tuned to work under optimial conditions.

EG: you might have a battery that's supposed to ideally function in environments of -20C to 100C. He would advocate to design the battery to operate between -50C and 150C to create a wider (ab)use case scope to cover clients really mistreating the thing, then tune it to work better between -20C and 100C, and then optimize it further to work best in the most common scenarios clients would use it in (let's say most users of the battery will use it between 10C and 50C).

You have to decide if that's useful for projects or not. Often programming companies will get slapped on the hands for over-engineering a product. (EG: they need a simple calendar applet biult into it, but the person in charge of making it spends a lot of time making a super-robust calendar applet).

But, then, catch-22, you build the thing to spec (what the client wants), then find out they're using outside of the spec they gave you.

EG: they neglected to tell you they wanted to track email addresses for customers in their CSR app. So, their CSR's started using the 4th line on the address input to input emails. But, some addresses really do need all 4 lines (especially if they have a "care of" / c/o header line, or an apartment numbering system that needs it's own line). Then company complaines that the 4th line is a wonky email data tracking location. Well, yeah, no kidding.. it's being used outside the original scope of the design.

So, you need to sit down with whoever is complaining that the project has bugs, and figure out if they really are bugs from not meeting design spec, or from them (ab)using the project in situations not outlined in the scope.

3) based on the above... Having a good Business Analyst / Project Manager will make-or-break a project. The BA may have simply done a crap job of scoping the system from the start. You're making up test scenarios per scope they gave you, system tests fine.. but you find out later that while you were making up tests for a bicycle, customer actually wanted a motorcycle and complains why there's no throttle and what-not.

So, I would go back to your system specs and see if you really created use-case scenarios to meet all predicted TYPICAL use-cases.

EG: if there's a date entry field... you set up a test to make sure it's only accepting dates. That's pretty no-brainer. You setup tests to input strings and numbers and what-have-you. It's scope as an optional field. Then customer comes back and complains that the field is requiring input.. duh, it wasn't scoped as a required field, so...

If you setup use-cases and tests per project scope, then there is either an issue with the BA scoping the project from the start, the customer not being forth-coming with the scope needed from the start, or someone customer-side trying to (ab)use the system in a way that's not intended. Possibly all 3.

So, go speak with the client to see what they're complaining about, and see if they're off the reservation doing weird things with the project that it was never scoped to do.

Write down all the issues they're having, then assign some kind of "likelihood" factor to each. This gives an idea of how likely someone would be in doing that action. If it's a very low likelyhood action, then chances are higher you probably didn't test it as well.. b/c the cost-benefit comes in testing the 20% of features 80% of people use, not the other 80% of features 20% of people use.

Basically, a client will expect perfection. That can happen over time, especially if it's an agile programming environment. (IE: every iteration will add new features and fix current bugs, so client complaining just needs to give feedback and wait for next roll-out / update.)

But, nobody can be perfect. A QA tester is paid to cover the most common scenarios, like a driving game where the driver can drive through trees... oops, that's a bug that needs to get addressed. But, someone finding some exploit where they can press the brake and gas and left-right on the controller at the same time (which shouldn't be possible, but the keyboard allows them) in order to cheat and instantly accelerate .. who the heck would find that in QA?

So, talk to the folks and figure out if it's valid bugs, or if it's "you guys have too much time on your hands and are coming up with obscure exploits" type of stuff.

Your company can't spend infinite amounts of time/money on QA, so you can't find all situations. If you tested the product to work within spec, then you should be fine. If they're using it outside spec, then that's an issue the BA needs to take up with them.

If you're not doing a good job testing within spec, then you need to sit down with the BA to figure out how they can help you come up with use-case testing better. (Soemtimes a BA will write really obscure specs or use high-level jargon that makes no sense to programmers, or will write a spec from customer's perspective instead of doing their job and translating into tech req's).

Basically, not trying to say you didn't do you job, but not saying they should treat you liek a scape goat either.

When anyone questions my work, I go to the source to discuss it with them to see if we're comparing apples to oranges, and often find out they're doing something completely off the reservation.

Not the answer you're looking for? Browse other questions tagged or ask your own question.