I am a Junior software QA specialist, I am kind lost to understand what to do in this scenario?

Question

Scenario:

Suppose that there is a mobile application (Hotel Booking). A particular use case does not work as expected (view booking history), as it has several defects and the business requirements need to be revised from a quality standpoint.

I would like to present to the management an analysis report that shows all types of issues (business requirements, defects, logic) for this use case and provide a possible solution for them.

What do you call this report?

Is it my job to do it as a QA specialist do it, or only business people?

Is there any supportive guide to create this report?

Answer 1

Wonder why you want to present this to management? What is your goal here, what do you hope to achieve. Do you want to show them they have unknown quality issues?

Normaly I would expect the product to have either an issue tracker or a backlog. Just put the issues on the list, discuss it with the business owners and let them prioritize. Together consider the context, what might look like a defect to your mental model, might not be an issue for the current userbase. Consider the business effects of the change, dont just swarm a development team with trivial defects.

I have never had the need for such a report, nor would I like to create reports like that. I do like to have the right quality conversations. For this I might need metrics over time. For example number of defects in relation to the number of features develivered.

Answer 2

You state you are junior. I wonder if you are a "team of senior + junior QAs and you are one of the juniors of the team" or you are "alone in charge of this".

If it's the first case, don't do any report, just tell your senior QAs teammates what you see and ask them for advice.

From now on for the rest of the answer I will assume that you are in this latter case and you are "in charge of all" without knowing too well what's the best way to add value to the business from your position in an organized manner.

So here's my advice:

A) Have an issue-tracker acting both as a bug-tracker and feature-request-tracker.

Making this resport you say manually is an real overkill and it will grow indefinitely until it gets unmanageable. When you have more than 500 bugs + 200 feature requests in the queue, you will not be able to manage it.

Considerations:

1. Every software (even small ones) needs at least a "bug tracker". There are a bunch of free ones there. I've been using "mantis bug tracker" for the last 17 years in many different companyes. It is really very ugly but it's free. https://www.mantisbt.org/ I'm not affiliated with them, there are many many others (redmine, jira (which is not free), bugzilla, etc.)
2. Each organization determines if the "bug tracker" also acts as a "feature-request" tracker or not. Most small companies do, as everything is simpler. This is why probably "issue-tracker" is a better name than "bug-tracker". A new feature can be seen as an "issue" and a bug as another "issue".
3. When you use a bug tracker as a feature tracker, you can "indicate" if something is a feature or a bug in a tag or field.
4. If that's a bug, you usually also specify other types of bug type (minor, major, whatever).
5. Usually you also specify the "project". Say your mobile application is a monolyth: you have a single project. But you can (if you want) separate "conceptually" in 2 parts: Frontend project (you set there colors, typos, button positions, responsiveness, etc...) and Backend project (you set there errors about data, backing services not responding, processes, etc.). That is arbitrary. If you are new to this, I'd advice to only use ONE project and after half a year, the "splitting" will come naturally to you.
6. Issues are auto-numbered. If you find 3 things that go together, open 3 issues with different number and set a mark to relate them together.
7. For the QA team, the bug-tracker is the "main tool" to track and document everything. It's the "always open tab" that you want in your browser.
8. For the dev team there are 2 approaches.
   1. If the company uses scrum/agile/whatever method, they will work on a "backlog". If the features are kept also in the bug-tracker (ie: the issue-tracker) the backlog will most probably say "for this sprint, in order, issues #345, #22, #177, #93
   2. If the company has 1 or 2 developers that "just do what's needed" then the simplest way is to use the "priority" flag of the issue-tracker (which typically will correlate with the bug's importance, so major bugs will most probably higher priorities than minor ones, but not necessarily). In this case, the coders just open up the issue-tracker every morning, scan the list for the "highest priority" and work on the first one. When done, the second one of highest priority, and so on. When they have resolved all the issues in "high priority" then they scan for "mid priority" and so on.

Here's an example of the Mantis I'm currenlty using, displaying the bug-levels. You can see there's the category "feature" and several levels of bugs:

B) Define black-over-white the bug levels.

If you watch the previous image, there are several names that could potentially cause confusion. Is "login not working" major or minor? Well... if noone can log in, it's clearly major. If you discover that potentially a new user registers with an email longer than 254 characters later cannot login because it exceeeds the maximum email length and that was not controlled in the registration process, but currently 100% of your users have emails shorter than 254 chars, then that is clearly a minor.

So, considerations:

1. It is task of the QA team to write a formal definition of those bug levels.
2. If you don't know where to start, I'd suggest this:
   1. Blocker => This is now breaking business and we are loosing money right now. For example 20% of users on an ecommerce "today" can't login because of an update we did yesterday.
   2. Crash => This is making the application to break. In a long term process (C, C++, java, etc. application running on a computer) this means the process ends with a core-dump. If we are talking about a web application this means a 5xx server error.
   3. Major => This means that any important process cannot be completed, although we are not looing money now (otherwise it'd be a Blocker). Example: In an e-commerce the user CAN pay AND the payment is completed but after that the user is not receiving any confirmation email AND also cannot access the member's area to see his payments.
   4. Minor => The means that any process (important or not) cannot be completed but there are workarounds that (with less comfort for the user) allow the process to be completed. For example: A list of providers in X country accessing from the country page is not working, but going to a general list, I can still get a fill list of providers and later on filter by country.
   5. Tweak/Text/Trival => Anything that "needs" to be corrected but does not break any process. For example a typo. This is an example of thing that is "low in importance by nature" but the business managers might set as top-priority: What if you are selling white boards and in a filter visible to end-users instead of "Filter by size" it says by accident "Filter by sieze"? That must be corrected because of the bad impression to the users but still it's not blocker/crash/major/minor.

So... does not matter if you take this specific definition or you define your own. In a company I worked in, we also had sub-divisions: Is this making us loose money MORE than 2.000,00 USD per hour, or 1.999,99 USD or less? This of course depends on the company. There are tiny-projects that just make $2.000 in the full month.

So it is YOU that need to write a definition. If you can, instead of closing the definition, make a "proposal" and have managers to approve it.

C) Let the coders resolve but not close. Doubly-link the issue resolution to the commit.

Having said all this, most issue-trackers suport "workflows". When you open a new issue it is in the "new" state and when a coder starts woring on it "captures" that issue so the other coders know that is being worked on.

When the coder commits the resolution, ask them to do this:

1. In the git, set the commit message to "Resolves issue #456". After committing, they will get a commit hash.
2. In the issue-tracker, when resolving, set the resolve message to "Resolved at project-name:6a5e3f47a82c56a23b4c6ae75e84f6f75a86c72d

If that's a multi-commit case (for example the coder touched both model and controllers and views and does 3 commits, one for each semantic group, use the commit hash that "contains all"

But hey, they resolve but do never close.

If you don't have Continuous integration and automatic tests (which by the nature of your question, I assume you don't) then do not allow the coders to make the merges. Their work ends in the last commit of the issue-branc or feature-branch with a merge-request.

D) Who closes then?

You or business.

To decide, ask if this was a bug or a new issue.

D.1 - Bug

If it was a bug, then you close.

Your daily-basis task is open the issue tracker and scan for issues in "resolved" state.

Then you git clone the branch and fully-test it. If the branch passes the quality control, then it is QA that merges the issue-branch into master when approved and makes the deploy.

Of course this holds only when you don't have auotomated tests and continuous integration.

Once merged, close the bug in the issue tracker and annotate the merge hash.

D.2 - Issue

If it was a feature most probably you have some kind of "demo day" every few days or "acceptance event" by the business managers.

If you use scrum or scrum-like things at the end of the sprint you'll do a demo to the Product Owner. In that meeting, annotate the numers that have been approved by the PO.

If you don't use agile, and it's just the boss saying "do this, do that" it is worth establishing some kind of "demo to the managers" so they can "test the feature" before going to production (so the branches are not merged yet in the demo to selectively accept or reject different set of features).

In either case it is "managers" that give the "close okey" to the "resolved" state of the coder.

You always go to the demo with pre-tested branches by QA. If you don't approve the quality that's not being demoed. I mean, the demo is a "request to merge to master", not a "let's play together to test if it works".

Annotate the numbers they approve and as you previously tested the branches, you merge them into master and deploy.

E) Your report... now here it comes your real anwer

With all this in place, do not do any report. Just open in the issue-tracker a non-privileged user that just can "watch" but cannot edit. Give that user to your managers.

Meet with them, ensure this:

1. The managers can open the tracker,
2. They Can login,
3. They can browse to the main issue-list,
4. They make a very-easy-to-find bookmark,
5. They save their password in the broswer.

So next time it's a mater of "1-click": They'll click the bookmark with the saved credentials and they'll see the list.

THIS is your report: The list they see in the mantis, in the redmine, in the Jira or whatever.

F) Make them easy - ONLY use bug priorities and avoid textual other that the title.

How do you communicate to them?

Avoid any text oriented to the managers. All the free-texts are a conversation between coders and QA. Except for one. The title. That must be also understood by the managers.

But give the title a very very very (x 100 times) very expressive name.

Consider the difference between those two titles:

Login failing again after update

versus

Login fails with google-OAuth because the token was invalidated by the deployer script

This raises something: QA is not limited to see "what fails". It also emcompases to hack until discovering "why it fails", so the QA person needs to have a hacker-mind and really try to "break" the system with a simulation of bad intentions. The QA people need to be hacker-minded... but that would be off-topic.

So in short

If you are in a team with other senior QA, better ask them.

If you are alone and have no senior QA besides you follow this:

1. Install an issue tracker. Up to here not-opinion based. If you don't know what to use, I'd go for mantis, but this is opinion-based and redmine fans would kill me. Let's leave this in "have an issue-tracker".
2. Define bug levels written.
3. Set aaaaaaaaaaaaaall your currently-known issues in the issue-tracker.
   1. If you treat bugs and features the same way, use the bug-tracker as a feature-tracker. Definitively if you manage less than 200 issues counting known-bugs and features go for this one.
   2. Alternatively if you have a different workflow for correcting bugs with respect to feature-requests, then set the bugs in the bug-tracker and the features in another place. For example trello-lists are very easy to manage and prioritize, to create product-backlogs and sprint-backlogs easily. Or if they have the same flow but you have 500 to 1.000 pending features and 500 known bugs. Then consider if you want to separate into 2 tools.
4. Have users for the issue-tracker (one user for each coder, and one user (read-only) for each manager.
5. Ask the coders to "grab the issue" when they start working with it.
6. Ask the coders to never "close" the issue. Mark it as resolved instead.
7. You, as QA, mark it as "closed" when you have tested it before deploing and give green light to the merge deploy.
8. Ask the Product Owner (or the business mangers if there's no P.O.) to close the ones in the "demo" of the new features that you have already tested.
9. Merge all closed bugs and features.
10. Do a final test after-merging and pre-deploying just in case.
11. Deploy.

If you don't have any time-schedule, even if you don't have any agile process, I suggest to do this in cycles. If you don't have any absolute reference, do this weekly or by-weekly.

So if you choose weekly, tell your managers you need from them full-attention 2 times per week: First-time on the monday 1 or 2 horus, and any-time on the friday 1 or 2 hours.

Each monday you'll meet the managers for a pseudo-plani

1. Open up the issue tracker with them together.
2. Review the list sorted by "nature" (not priority) (blockers, majors, minors, etc).
3. Review what priorities were non-correlated to nature in previous weeks and see if thay still hold good. For example:
   • a blocker that business said "we know we loose money here but we don't care as there are more important things to do right now" (may seem silly, but can happen if you loose but not too much (say a 2% of sales) and the managers need new features to close a deal with a partner that will boost the busines half-term.
   • a typo or minor that for them is very important
   • Correct the priorities in real time if needed.
4. Finally review with the managers the "priority tags" of the issues recently open.

Once this is done, double-check sorting by "priority" (not nature) and agree "we will work this week first* on this, when we kill all those **then on that, when we kill all that then that. We'll work in this order until the end of the week, see how many of them we can do.

Each firady, meet the managers for a demo

1. Open up the issue tracker and list the closed bugs and resolved features.
2. For the bugs show the corrected version and say "this has been resolved, and as it was a bug I already closed it"
3. For the features say "so here's what we did this week, I have already tested and for me it is ready to go to production". Show the feature and ask "merge?" and get a yes/no (it can happen that a feature is ready to merge but managers want to keep it for a later release).

With all this you do not need the report you were asking for, but instead you'll add much more value as the list will be a "live list", not needing to be hand-crafted every time. The list will reflect the "real work" in "real time" if they enter to see it ad mid-week.

You have thousands of screenshots over there: https://www.google.com/search?q=mantis+screenshot&newwindow=1&client=firefox-b-d&sxsrf=ALeKk03RLE002YMYHG0etua8TAmcPaSadw:1592050273735&source=lnms&tbm=isch&sa=X&ved=2ahUKEwiR7J644f7pAhWNlxQKHc5oBZkQ_AUoAXoECAsQAw&biw=1760&bih=869

An example of "a prioritized listing" could be this one (this is a real example from the ones I manage):

Answer 3

Is there a possility of financial loss by the end-user and that loss that is significant?

If so, advise management.

If there is no loss, raise it as a functional issue and prioritize it's importance against other issues for the product.

Answer 4

Testing is a risk analysis and mitigation activity. You mention a specific use case is failing.

Consider:
- how likely to occur is this use case? And what would be impact be if it did?

From what you've stated, it sounds like the user would not see any booking history. Is this the most typical user type or an edge-case/fringe user? Is the history relevant to a future booking/purchase (i.e. does the booking history form part of the future booking userflow)?

You mention it has several defects. Is the feature totally unusable, or inconsistent? Can you repeat the bugs on demand?

Personally, I would not write a report, but it you did I would expect it to be a Project Quality or Defect Report. These are typical written at end of a testing phase or build cycle. Or most typically, at the end of the project's testing window.

Armed with the contextual information you analysed earlier, I would seek to discuss the issues with a developer and business representative to ensure the understanding from the requirements (which sound ambiguous) is correct, and for all to get a better understanding of the size and complexity of the issue.

If this needs to be escalated to Snr Management, it's better that it is presented with a coherent explanation of the problem, impact and most importantly, the possible solutions. This cannot be achieved by QA alone, and should be managed at project/Team level with your valuable input.

Answer 5

Turn it around.

Instead of viewing it as

A particular use case does not work as expected (view booking history), as it has several defects and the business requirements need to be revised from a quality standpoint.

view it is

A particular use case does not work as expected (view booking history), as it has several defects and the quality requirements need to be revised from a business standpoint.

I know it is small and subtle but it is actually a big change. Your direction should come from the business.

Now. :)
I know (from experience) that the business may be:

• unaware of the issue
• unaware of the consequences
• accepting of the consequences.

If you feel that they don't 'get it' then your job is to educate to the point where you've convinced them of the logic of your argument. Then they will push for these requirements, not you. You will just verify that these requirements have been met. Requirements still exist in Agile btw (in case of push back). You can use BDD and Cucumber to document them as living documentation backed by code quite nicely.

Answer 6

In every software testing company 'Defect Analysis' report is used to analyse the issues.

If team consists of a single QA member then this is the responsibility of that particular member and if multiple members are involved in QA team then majorly this is the responsibility of QA Lead of the project.

With the help of this report, QA can showcase the other product members about the kind of issues encountered in the product wither it was related to the requirement or logic is not properly implemented.

Every QA team follows a process to have a bug tracking tool like Jira, bugzilla. You can collect the data from the bug tracking tool.