Should QA report bugs on features before they are fully implemented?

Question

I am a developer and I generally check in partial implementations so that I don't have too much uncommited code at any time.

I make sure that the application compiles and that it does not interfere with other parts of the code. Only when I have implemented and tested a feature do I mark it as "done". This is when I feel that SQA can formally start testing the feature.

Lately our SQA has been told to start testing things and logging bugs that are not fully implemented yet. To make matters worse we have 3 SQA and they all log the same bug.

This increases the number of bugs that get assigned to every developer and then not only do the developers have to complete the task on hand but they also have to work through all these bugs that have been logged.

For me personally when I get 20 bugs on any feature I feel as though I did a sloppy job.

Is this a normal/accepted practice?

Answer 1

It is always a very good practice to start testing as early as possible.

By not fully implemented if you mean still under development, then I suppose that is not such a great idea. Yes, you can go for incremental build releases for testing, that might be helpful.

When you say that your testers all log the same bugs, well there's your problem!

Do you maintain a central repository for tracking bugs?

A bug tracking system of some sort? If yes, then do your testers check whether a bug has already been logged before they log their list of bugs?

Do they not coordinate among themselves?

There seems to be a serious communication and understanding issue.

Answer 2

I think the problem arrives from how you use your VCS:

You want to commit your code because you have finished a sub task and want to have it backed up in case of e.g. a hdd crash on your workstation.

Your testers see changes in the repo or get an automated build which they test.

One solution could be a separate working branch which is work in progress. No tester should test this code unless asked to do so.

When your feature is finished you can merge the code into a production/testing branch in your VCS.

That commit can be tested and bugs can be filed for all builds from that branch.

Answer 3

To me this is a perfect example of what happens when effectiveness of testers is assessed based on the wrong KPI, in this case, a number of bugs reported. In the end it will harm testers, developers, relationships between tester and developers and as a result product quality.

To me this is wrong KPI and there have been at least two discussions here why is so:

• What is a good KPI for software QA?
• What are the most useful KPIs for tester

I, personally, sometimes report problems found in incomplete work, but I ask before reporting it formally. I ask whether it's because of incomplete work or just requirements are wrong and we need to discuss requirements or plan fixing it in the next sprints.

Joern Boegeholz suggested also to work in branches. This is great approach. However, you said you were working for two weeks on UI changes and it has not been finished. To me this demonstrates that perhaps your stories might be too long and you should try to work as a team on defining stories that are smaller and of finer granularity. That could be kind of compromise also between testers and developers: they could get things to test more frequently and you could focus on your work until it is complete. Finally, merging shorter branches might be less painful.

Answer 4

Most of the companies I've worked for over the last 10 years are using an Agile approach. In this approach there are several factors to pay attention to such as:

• favor direct communication over formal processes
• encourage a team approach
• sit together

I find that in such environments you want to seek out practices such as:

• pairing on test plans before development even starts
• talking about how something will be tested before development starts
• pairing on features at several stages through development
• dev's pairing for qa work, testers pairing during dev work, 2 sets of eyes

When you wait until you feel the feature is done enough to be tested you run the risk of more effort / not wanting to change to code already written and the risk of defending what you just created.

This does require more communication so that folks are aware of partial implementations, issues to ignore, etc.

Answer 5

It makes zero sense to waste time on testing a feature (and reporting bugs, and triaging them) if development is not finished yet. All this work will have to be redone again when code is finished. Something is strange in your process, if testers have so much time that they create extra work for themselves like that, which has no benefits to the project.

Do your testers have separate environment for QA testing? Or developers deploy code directly from DEV to PROD? So QA is forced to test in DEV, trying to find bugs before they are in PROD? Separate testing environment would prevent that.

Syndrome of even deeper problem is fact that all SQA will enter the same bug. It should be QA personnel responsibility to triage the bugs and see if there are duplicates. Especially if all bug reports are related to the same newly developed feature under testing.

If your process includes QA environment, such situation can hardly even occur, because code is deployed to QA only after development is finished.

Edit: If you work agile, and have resources, you can start exploring/writing tests for parts of the app which will not undergo rapid changes (which "settled"). But such tests are just preliminary, and have to be repeated when development is finished.

In the OP's situation that makes little sense, because (from his question) the lack of the communication between QAs, and between QAs and developers. Agile development depends heavily on communication.

Answer 6

No, it doesn't make sense to test code that you know is incomplete and not ready for testing.

But rather than blaming your QA folks... I suggest that you take a closer look at your software development process. It sounds like there is a failure of communication between developers and QA, and a mismatch of expectations.

For instance, it sounds like you are committing your incomplete code to your shared code repository. Instead, it is better practice to use branches. You should be doing your development on feature branches. For instance, here is one sample workflow:

• You have a master branch, which holds the latest version of the code. You only merge code into this branch when it is ready to go.

• For development of a new feature, you create a private feature branch, which you use as your write your code-in-progress for the feature. Once the code is done, you can merge it to the master branch.

This makes it easier to integrate testing and SQA into your workflow. For instance, once you think a feature branch is ready to go, you merge it into the master branch; the QA team focuses only on testing and evaluating the master branch, so they never waste time testing in-progress code that you know isn't ready to be tested.

Or, once you believe the code in a feature branch is ready to go, you can send it to the QA team for them to test. When it passes QA, it can be merged into master at that point. There are many alternatives and variations, but the point is that private branches enables better communication: it helps establish shared expectations about what code should be evaluated and what code is pointless to test.

And ultimately, that's what it is all about: communication and aligning everyone's expectations.

For more on this kind of workflow, see also

• To branch or not to branch?

• My coworker commits and pushes without testing

• Dev vs Stage environment vs Prod Environment

Answer 7

The fact that multiple testers test the same thing without even talking to each other shows very clearly that the process is broken. You shouldn't have 2 developers develop the same feature independently and you shouldn't have 2 testers test the same feature independently . It sounds obvious, but if 2 people work on the same thing, they need to know about each other's work and work together.

You cannot test work in progress if you lack strong communication channels between tester and developer.

Let's contrast this with a working setup I used a while ago:

The tester is part of the daily meetings where everyone takes a minute (and not a second more) to let everyone else know what they're working on, and what has been completed since the last meeting. The tester is part of the meeting and lets the team know if they have time available to test some unfinished things, because testing finished features takes priority. Similarly, the team lets the tester know early if some finished features are likely to be touched again by refactoring, and therefore will need to be re-tested.

If a developer has some half finished work which can already be partially tested, and the tester has time to test, they both mention it in the meeting, and then sit together after the meeting to figure out the details.

The advantages of testing partially complete features are twofold: First is that the developer still has everything in his head and therefore can fix the issue spending less time than he would if the defect were raised later. Second is that by detecting and fixing the bugs sooner, your product will be finished sooner, i.e. the build-test-fix-retest-release pipeline becomes shorter. The disadvantage is that the tester needs to test more often. You sacrifice tester time to speed up time to market.

Answer 8

You wish to check your code in to keep it save. The testers test everything they can see, maybe because it is not clear what is not finished, or maybe just for another reason. There are a few actions you can take to stop this problem.

• You could use Feature branches
  • You only merge the code into “main” when it is finished
  • Other developers can see what you are working on if they wish
  • But you have to merge in the code when done, this can take a long time
  • And other developers are not working with the same code as you until you do the merge.
• You could use Feature Toggles
  • You have a toggle, maybe runtime or compile time this enables the new feature
  • When the toggle is not set, the new feature is not show up in any menu etc
  • If it is a runtime toggle, then testers and managers can turn on the feature if they wish to look at it.
  • But as the feature is off by default, testers are unlikely to raise bugs against it.
  • If when you come to ship the feature is not working well, it can be turned off for that shipment
  • It is not hard to licence the feature on a per customer bases if you wish
  • But you have to put in support for enabling menus based on the feature flats etc.
  • Once development is finished and tested, you can decide if you wish to leave the toggle in the source code or remove it.

See the question “Feature Toggles vs Feature Branches” and Martin Fowler’s blog post on FeatureToggle.

Answer 9

I'd say the issue is in the complete process.

In your process, you should have a delivery document which explains the SQA team what have been developed, and what is still in development. Using this document, the SQA won't test anything not developed yet.

If they absolutely must (due to management constraints) test in advance, you could cut your functionality in smaller pieces, and "deliver" some parts of a functionality only, which could be tested while the complete functionality isn't finished yet.

The fact of cutting a functionality into atomic pieces really helped us testing the right things, without loosing time in test on unfinished parts.

For your "same bug log three times" issue, this is the responsibility of the Test Manager, who should organize his team so that they don't test the same functionality at the same time. It's the best way to loose a lot of time, both for the developer and the test team.

Answer 10

Is it normal? Yes and no. I would say that first and foremost if you are in an AGILE environment, the product owner or someone else should be scrubbing duplicate defect reports before committing them.

Also, testing tasks should be presenting to QA once a feature is working, not typically before. That just wastes precious QA time and makes the backlog messy.

In the end, though, I would avoid checking in incomplete code unless you need to because someone else needs to make changes. Even then, it's difficult for multiple people to be working on the same code. If another engineer doesn't need to alter the same code you're working on, don't check it in until it's complete.

Answer 11

In fact this is, to an extent, quite normal if you are using test driven development, TDD - you can expect near 100% of your tests to fail at the beginning of development and things to improve as you go on.

Possibly you need to work on your companies workflow so that failing tests are logged as failing tests until you attempt to mark a feature complete - when you do then test failures become bugs. Feature branches are a good way to do this.

If your testers are duplicating bugs then there is a serious issue with them not checking for known problems and that needs working on by your QA manager.

Answer 12

As a tester, I love exploring partially developed code, and giving early feedback to the developer.

The key thing, as others have mentioned, is strong communication between developer and tester. I need to know what areas are likely to still be incomplete, so I don't waste time trying to isolate an issue. The developer needs to be prepared to outline what's finished, and perhaps discuss some tips on ways to workaround unfinished areas (for instance, if creating an order via the UI isn't finished yet, I might want to insert data directly into the database so that I can still look at other areas).

The biggest benefit of this is that my understanding of what we're building grows alongside the developer's, and I can feed in my questions about potential edge cases as they're building, thus saving them valuable time and deepening both of our understandings.

Some other answers seem to assume a test case based approach, where you run the same test cases time and time again. Frankly, if I was running exactly the same tests once the feature is done, I'd consider that I hadn't learned anything from my previous work, and would be worried. That early testing stage is where I develop my mental model, and think about test design.

That may seem a little long - but I wanted to describe how it can be done well. Your situation is clearly not being handled well:

• communication via bug tracker
• KPI driven testers attempting to raise the max number of bugs
• many duplicates, lack of communication between testers
• attempts from you to clarify scope totally ignored

to name just a few points. I'd recommend discussing with line management to understand the purpose of requesting that QA get involved earlier, and explaining the additional overhead of running everything through the bug tracker. If possible, if there is a particular tester who you respect, you could suggest that it would be more practical to assign one tester at this early stage, and that would let you work together to develop a process that works better.

Answer 13

Firstly, don't be put off by committing your code, you should continue to do this early and often.

Any code that goes into the master branch I feel should be ready to be tested straight away and all of the automated tests should be passing at this point.

QA should be aiming to test your work as soon as possible and not leaving everything until the end of the sprint, but at the same time they shouldn't be testing something that is not yet ready. This is not only a waste of their time but will likely create a division between the developers and testers.

I would try suggesting something like this.

• When you have a feature that is ready for QA, inform them and schedule a short demonstration of the feature. This should be where the developer sites down for 5 minutes with the tester and demonstrates the new functionality. This should not only improve the relationship between QA and the developers but often can also identify issues during the demo which the developer can immediately work on fixing before any tickets are raised.
• Your QA team constantly monitor the master branch and report any defects found in this straight away. So that they are not all reporting the same issue at the same time, maybe each one could do this a week at a time. This will also encourage developers to keep features that are not ready in their own feature branch.

Answer 14

Well, that's actually like Dev Box Testing. Dev Box testing helps ensuring that an extra level of verification is made. It ensures the quality of the user story before it is committed to the QA environment. That said, there shouldn't be JIRA tickets or tickets raised for the Dev Box Testing. Instead the bugs must be reported in a sheet or over the email. If you're hell bent on using JIRA then there can be a single DEV Box Testing ticket that has all bugs mentioned in it in a brief manner.

QA is always a quality thing and not a quantity thing! Number of bugs don't always reflect the quality of testing.

Answer 15

If the feature is delivered via consecutive stories with well defined DoD then QA should test those stories as soon as they reach "ready to test" state. As @PeterJackson mentioned "It is always a very good practice to start testing as early as possible.", so QA sould follow feature development with test of stories handled by developers to testers. The crux is for all team players (ie. analysts, developers, testers etc) to understand feature(s), context (like the current SDLC phase), DoD's and the direction the project is heading. Then testing and reporting issues early increases the chance to deliver on schedule and what is needed.

Answer 16

This sounds like the feature is scoped very large and not done incrementally. Increment the feature in segments, deploying each segment into the production environment. Make sure the testing environment is tested thoroughly before other deployments are made.