Bugs keep reappearing in our GUI software during development. How should this be addressed?

Question

We have a software product has a lot of usability features, and we're constantly changing it. We have issues with things like persisting scroll position which seem to get fixed, but then are broken again when we release. We have good testers, but we seem to miss things like this because there are just so many features in the application it's hard to perform thorough regression testing.

How is this problem best solved? I'd love to have automated UI tests, but those are fragile. I'd love to have our testers test every feature every time we implement a new feature, but that would take forever, and seems impractical. I wanted to gauge how other software shops do this and how they avoid recurring bugs.

Answer 1

You need to understand what are the core issues with these repeated breakages before you decide how to fix the problem.

Relying on (more, improved, better, faster, etc) testing may not be the most effective solution here. It might not even help at all.

• Do you have code merge issues? Are fixes becoming undone because of that?
• Do you have new bugs appearing, with similar symptoms?
• Are these UI features basically not fully "cooked"? Perhaps you are wasting time testing them too early in the development process?
• Are developers making the same mistakes in new code? Are they well trained?
• Does sufficient documentation exist?

Lots of potential core issues here - each with different solutions.

Answer 2

It is hard to draw reliable conclusions about an organization from two paragraphs in StackExchange, but it sounds to me as if you're going too fast. The fact that you are constantly changing the product suggests it is fairly new. With new products and new companies, when most of your users are early adopters, it may be more important to release quickly than to release with high quality. If that's really where you are, you may have to live with that condition until you have the luxury of more time.

You referred to UI issues in the title and twice in the question, so I assume your quality issues are specific to the UI; i.e. the rest of the software is in better shape. If you have the time to spare, I suggest slowing down and taking measures to ensure the UI is more thoroughly and consistently tested. If your testers don't use a test plan, someone should write one for them, at least for the aspects of the UI that are especially buggy. Are your testers using the same kind and variety of environments as your users? For example, if you have a browser-based UI, are your testers using same variety of browsers and screen resolutions?

You might investigate why your developers are creating a lot of UI bugs. Are they aware of the quality problems? From their perspective, is it more important to go fast or to unit-test their work before they check it in? Are the developers using the right infrastructure? If a change in one thing causes a cascade of bugs in other places, you may have a design problem.

Answer 3

If your only concern is layout bugs in the UI; you can give fighting-layout-bugs a try. Its an open-source java library that checks for layout bugs. You can find more info here

http://code.google.com/p/fighting-layout-bugs/

In addition Selenium can come handy in creating a automated regression suite. You raised a question of these tests being fragile; which is true to an extent. But if done neatly they can prove to be an asset in the long run. You need to balance the automation and manual exploratory effort to get the best of both the worlds.

Gojko has described this beautifully than I ever can, so here's the link

http://gojko.net/2010/04/13/how-to-implement-ui-testing-without-shooting-yourself-in-the-foot-2/

Answer 4

Short anwer: Develop an automated regression suite, UI tests are not fragile when done right.

Longer answer:

I would suggest to invest time in developing an automated regression testsuite using selenium. A lot of times people say UI tests are fragile but in my opinion they're only fragile when they're not programmed right. It is true that it is hard to create a well functioning testsuite but in the end it is worth it.

Think about how much time your testers will spent roughly on executing repetitive tasks until the end of the project and think about how much time it will take to develop a well functioning/maintainable testsuite to get rid of those repetitive tasks.

If you choose to do create an automated testsuite you or your client will end up with a system wich is far easier and cheaper to maintain. A well developed automated testsuite is worth a lot of money. While manual testing is more like throwing away money.

You're team now has a huge technical debt because of the fact that there's no automated regression suite. You should figure out if it's worth the effort to remove the debt right now and if so go ahead and remove it. Try to make everyone involved understand the ROI of developing/maintaining such a suite and try to prioritise development of such a suite along with the prioritization of the rest of the work.

It's not always easy to convince management that it might be a good idea to use a lot of time to develop a testsuite instead of using that time to develop features but I think you should try anyway.

It also might be a good idea to hire a selenium consultant/trainer for a while to train your testers/developers on how to build a well developed, well functioning testsuite (if your company doesn't have any experts allready). Developing a good testsuite is very hard and experts could prevent you from walking into all the pitfalls.

Answer 5

How is your check-in process/policies?

I've found in the past that development peer reviews coupled with test peer reviews have helped a lot to reduce regressions such as this. A checklist can be compiled identifying the top 5 to top 10 riskiest areas and set a rule that if any of these break on a developer's machine (provided they have the latest code loaded), they are not allowed to check in. The checklist should take no longer than 5–10 minutes to run. These tests can be manual or automated, depending on how much those screens change.

Yes, it will add to the check-in time but this ensures that you catch those bugs earlier and fix them on the spot rather than spend the time testing, raising bugs and developing it and testing it again.

And with this process in place, soon enough, the developers will learn to check these things before you even do the peer-test review. As a result, they will produce better code.

Answer 6

The few problems that your team is facing, and the possible solutions:

Keeping up / too much load - Either the company is trying to roll out things too fast or your product does demand constant feature changes or the dev team does not do their part of testing. In either or a combination of these cases, a lot of load falls on the test team to keep up. This may result into tester's fatigue or the development of the attitude that even if we try our best, there is so much that there will always be bugs.

Automation - The test automation is as fragile as the test team that develops it. If you have engineers who can develop good test automation, that will help beyond words. If you are not confident enough, start with automating tasks that are relatively stable and most repetitive e.g. if you have to test 'create user' each test cycle, it is worth automating. Automation will also provide a level of creative satisfaction to the testers and something to look forward to. And it will, most certainly, give you more coverage at a greater speed.

Communicate with other members on the project - Communicate with the project manager or someone of the likes that the test team is being overloaded and if they can reconsider the number of features being rolled out each release. It is also highly likely that the development team feels the same. As a result, they might not be able to test their code thoroughly before passing it on to the test team. The company has a choice - add more features with more bugs, or less features with better quality.

Test team - Finally and most importantly (if this is not already done), have the manager meet each individual team member at least once in two weeks and get their feedback. How satisfied they are, what do they think about their work, load, etc. Get updates on what they did since the last meeting and what they are looking forward to achieve before the next meeting. Any suggestions to improve the quality of the product. This will also give an insight on how good the team actually is, are they motivated enough, if anyone is not doing enough, and much more. You can also do daily stand-ups for daily updates within the QA team.

Hope this provides some direction.

Answer 7

Very Good Question. Thanks for asking it

Automation can address this issue if we plan to test production / real time scenarios. Below is approach towards automating the product.

Phase I

• Get started with Automation during development phase
• When you have the mock UI, Id's

Phase II

• During Testing ensure your automation is in usable state
• Add / Update Automation cases bases on bugs# identified

Phase III

• When the code is deployed in production, Capture from the logs majority sequence of user workflows

• example

  • 50% of users login, do a search and then buy a single product
  • 30% users login, look for offers and order multiple product
• Automation can be useful if we leverage it based on actual user scenarios from production logs. This way you can be sure you test all major user workflows

Functional Testing Effort + Regression Test Effort + Updating Automation Effort might be a challenge considering timelines. Possible solution can be Developers running automation suite and identifying bugs in earlier stage. Test Team can make it in scope of work to add production workflow (Major 6 workflows based on captured logs is a criteria for automation effort)

• Even if you cannot automate production workflow keep track of production bugs# and missed scenarios and ensure they are regressed for every build
• Have a Knowledge repositiory / Guidelines not as test cases but atleast to say (Areas where possible bugs can occur / Checklist ) would help
• Above all ownership the individual demonstrates matters. Learning from the bugs missed and adding relevant checkpoints to address it for future releases
• Accquiring domain expertise takes time but having good knowlege of product would help to do good exploratory testing
• Track the bugs, Check pattern of recurring bugs. If we can arrive at any assessment quality of dev in certain area/ certain developers need to be improved, take it forward as recommedation from QA Team

Answer 8

It sounds to me like you are not reusing your code.

I don't think you can run a comprehensive UI test automation with a complex UI unless you start early. It's still a good idea though! But I think details like scroll position are not your focus here. (Maybe we just did it wrong) In our UI tests we test use cases and some general smoke tests.

So maybe it's a design issue?! If you encapsulate the UI into controls and reuse them, they are less likely to break.

I know the instinct to blame testers, but as a developer I need to focus on creating fewer bugs.

So I use designs that encapsulate things that change and components that are reused. Decouple your design so you can reuse components that proved to work. For us TDD did the trick.

Answer 9

If it's impractical for the test team to thoroughly test the GUI, then see if you can get someone else to test it instead!

Found out what testing the developers do before they hand over the software and see if you can get them to do more / better testing. Alternatively, can you get your customers to test the software for you (i.e. Beta Testing)?