Testing an application under continuous development

Question

Ill try and set the scene for you all as best as I can.

I have been working as the sole Tester on a suite of 13 related applications which is the end result of 5 separate developers work. Basically we operate on a two week update cycle (was originally weekly updates) with the client providing us with new features to implement as and when they require them. We are currently using the 'Agile' methodology of a Kanban board to keep track of all the work items.

My problem is that I find myself struggling at times to keep on top of everything, a lot of the applications have dependencies on the other applications. So even when one application hasn't technically been updated chances are that it has still been affected by a change elsewhere.

Also because of the constant development on the applications it is very difficult to get a stable build to test. To counter this I have tried to introduce a sort of 'cut-off' period where no-one can merge to my test branch 2/3 days prior to a release (however this delay period doesn't strike me as particularly 'Agile').

Obviously I don't want to be doing complete sets of regression tests every 2 weeks just to ensure a tiny change to a database view somewhere hasn't broken a page load in one of the related applications. A while back I spent a fair amount of time working with Visual Studio's Coded UI tests in an effort to use automation to help but I found the tests it created to be very fragile and having to spend more time maintaining them than actually testing the applications themselves.

Does anyone have any experience in similar working environments and has any advice they can offer because this is getting to the point of frustration, obviously more hands on deck would be nice but that is not an available option.

Answer 1

Craig, I have been working in a similar environment for 6 years. DEVs and QA are on the same box, using the same code which is constantly in flux. DEVs check out a program(s), update & re-compile while I am in the middle of testing. At first, I was very frustrated as you are. When possible I do create my own "sandbox" (db only, not actual program) but sometimes it is not feasible due to the volume of new programs being added. Sometimes I get warning prior to my session tanking but usually I don't. A little background:

1. We have 2 testers, yours truly & 1 other.
2. We support 5 developers, occasionally one or 2 more.
3. We have 1 core tool that customers purchase with 12 optional modules, 5 versions of our software to support, 5 versions of the OS to support and 3 platforms to support (IE & FF are the only browsers).
4. QA verifies our own bugs and customers no matter what else is going on (bug fixes come all the time; there is no break like there when maintenance devs are pulled for new dev).
5. We consistently push service pack builds out around every 60 days, and new versions every 18 mos or so.
6. All this is done w/very little, if any, requirements.

Does this sound familiar yet?

For the record, I am not complaining; I am stating facts. This environment is working very well for our team. There are always things I would change/improve upon, as with any team. We have a very devoted customer base which speaks volumes for our system & it's success.

Based on what you have written, Craig, I hear 2 frustrations:

1. How can I possibly test thoroughly in such an environment? Builds are not stable and bug fixes affect the other modules and I don't want to miss anything.
2. Is the cut-off described the best solution?

To #1: Accept that you will miss bugs no matter how thoroughly you test. What our team has done and it works well for now is I developed a "mini regression" suite that touches all modules, testing only the very core functionality. I weighted and scaled it so in a time crunch the lower weighted areas are not touched. Much of this is legacy code with very few bugs.

To #2: I would suggest that during the 2-3 days of your own "code freeze" you ask the devs to be proactive on code changes & just tell you program x is changing. This will only work if they know it's only a few days. Don't worry if it's agile or not. Find what works best for your team and environment and go with it.

As far as automation: If it takes as long to maintain the scripts as it does to manually test it, then don't bother. We have zero automation because it would take 1 person full-time to manage it.

If I may, one final thing. Your sanity: protect it. Do this by submitting your dilemma(s) to your manager, responding as he/she suggests and then let go. You know you are doing the best job you can, going above and beyond. Find the positives in this environment, learn from them and hold on to them. You will use them today and in the future.

Answer 2

After reading your account, I have a few questions. They might sound harsh at first, but your account sounds to me like you're in an organization that has to consider some harsh realities.

1) Is it the complexity that's a problem here, or is it your reaction to the complexity? (Along with your organization's apparent lack of reaction to the complexity?)

2) Are your programmers required to write code that they can warrant will work, or are they merely required to write code?

3) Do you have any managers working there, or does everyone simply do what he likes?

A couple of specific points:

We are currently using the 'Agile' methodology of a Kanban board to keep track of all the work items.

Try not to confuse "methodology" with "practice" or "tool" or "technique", or even "method". Methodology is the study of methods. Now, that may seem like I'm being pedantic. That's not my intention. However, I have observed that people often use grand words to mask less-than-grand knowledge or adoption of a subject that needs continuous study.

My problem is that I find myself struggling at times to keep on top of everything, a lot of the applications have dependencies on the other applications. So even when one application hasn't technically been updated chances are that it has still been affected by a change elsewhere.

What might happen if, instead of thinking of all this as a problem, you thought of it as a test result--evidence of a problem that testing is revealing? The fact is, poorly-understood interdependencies represent a problem for the product and ultimately for the business. Since you don't have control over the code and the interdependencies, you can't change that--but you can sure as heck report it as a problem. Anything that makes testing harder, or makes it take longer, allows whatever bugs are in the product to hide out for longer, which piles on the risk.

Also because of the constant development on the applications it is very difficult to get a stable build to test.

This is another instance of the same problem. When you finally deploy a build that was stable enough to test, and you did you did a burst of testing under a lot of uncertainty and extreme time pressure, what makes you and the development team and the business think that it was stable enough to release?

To counter this I have tried to introduce a sort of 'cut-off' period where no-one can merge to my test branch 2/3 days prior to a release (however this delay period doesn't strike me as particularly 'Agile').

Don't worry; it doesn't sound like anything else that your group is doing is particularly agile either, except for the speed part.

One of the things that the Agilists (the XP people, at least) talk about is the idea of sustainable pace. Your testing is revealing, and you are describing, a pace that seems to me unsustainable. You could ask, "How do I cope?" But I think here trying to cope would add to the problem. A more important question is, "How is the development team going to respond to the information that I'm revealing about the product and the projects?"

Does anyone have any experience in similar working environments and has any advice they can offer because this is getting to the point of frustration, obviously more hands on deck would be nice but that is not an available option.

More hands on deck is an available option (although I don't think it would really address the problem). In any case, it's an option that your business has chosen to reject. What other choices could they make differently?

---Michael B.

Answer 3

With a 5 developer to 1 tester ratio getting the developers writing automated unit tests is long term strategy I would press for.

The question describes problems understanding the systems dependencies (and interdependencies). I believe adoption of unit testing will drive some behavior that will make those dependencies more obvious by design. It isn't the final answer though and it will take time for some fruit from that labor to be revealed.

To get some short term relief I would call some meetings to get the dependencies really documented. Visually. Include every component (dll), every database, every UI, perhaps even every script, etc. You can't include too much.

Answer 4

Let me start by saying you have a great list of responses by a group of highly experienced and respected testers make sure to use their advice wisely, I know I will :)

My situation is quite different, we are a medium sized organization preforming agile scrum methodology with many development and testing environments. That being said, we preform hundreds of builds and deployments a day. The key to the success of our constant state of development relies on Unit tests, build and deployment automation, and constant communication.

If I was in your situation I would focus on the following:

I can't even begin to truly describe how useful unit testing is to the success of continuous integration. That being said, this is something that is probably out of your control, make sure to push for it but its something you can't do yourself.

Looking into automation again. Considering you are an army of one, I would recommend getting a set of smoke/sanity tests put together to run on deployment starting with the most important app first. I'd really focus on something very simple that checks the app is up and responses to requests. Getting together a simple suite (maybe one or two tests per app) to run on every deployment will help you find problems sooner, a key to CI that you are greatly missing.

Do code reviews with the developers. I find this extremely helpful to create tests and understand how the code actually works behind the scenes. In addition asking the developer questions will normally find issues before you even start testing.

Interact with the business. Is your company happy with the current defects that seem to pop up every deployment? If they aren't, you may be able to push changes to improve code quality.

Finally let me say that I actually envy your situation a bit. As the sole tester, you can try out new things as you see fit and see what works for your team. I'm a bit of a control freak and would love the challenge of finding solutions to the problems you've outlined. For me, figuring that out is the cool stuff we get to do.

Good luck and happy testing!

Answer 5

I will offer some advice and then I will describe my own situation for comparison with yours.

You mentioned 'Agile' twice but said you are near frustration with your current process, which suggests that while you may not value your current process, you do value 'Agile' principles -- perhaps the ones described in the Agile Manifesto. It seems to me that agility is about using frequent communications to react quickly to change. If you cannot afford a full regression test for every cycle, you must find a way to test selectively, as you are doing now. Perhaps your developers could communicate additional information that would let you improve your selection process. If they do not do so now, they could suggest areas that may be impacted by a change. (This will help only if developers take this responsibility seriously, but in my experience most developers will do so.) Another option, as Rob Murdoch mentioned, is to improve your understanding of how components in your applications depend upon each other, so that you can decide on your own how to translate a bug fix into areas that require testing.

If you cannot hire another tester, and if there is too much work for one person to test, it may be time to push some of the testing back onto the developers. This could be a accomplished with automated unit tests. Another option is to ask developers to assist with manual testing. If you go this route, I suggest asking the same developer to test the same thing every time. Sometimes, after a developer has tested something manually several times, they will come up with ideas for testing things automatically. Or if a developer finds bugs in an area over and over again, they may think about ways to make the bug-prone parts more reliable.

Here is my situation. I am the only test person for a three-developer team on a single application. We do not intentionally follow an Agile process. We deploy a big release every six to twelve weeks, with smaller "bug fix" releases in between. After I finish testing new features, I need a week for regression testing.

For the most part, my developers do not write unit tests. I often have the luxury of time to write automation, so I have some API- and file-level automation for selected areas that tend to be buggy, are especially important, or which are especially time-consuming or error-prone to test manually.

You described an unproductive experience with UI automation. I am not familiar with Visual Studio's Coded UI tests, but my experience with other UI automation technology is that it should be used selectively. My experience, too, is that the maintenance cost of UI automation often outweighs its benefit. I only use UI automation in places where the interface changes very little but where bugs tend to arise due to implementation changes. (I use this principle for non-UI automation, too.)

Answer 6

I have to admit when I read this my first thought was "I didn't know you worked for my employer"

I'm facing similar difficulties as one of a team of six testers working with about two dozen developers on (currently) something like ten major new features against an unstable code base with a hard release date, complicated by immense structural changes to the software in test that broke ever single regression script (most of the problems were buggy software, too, not changes that the scripts exercising the GUI had to account for).

Automation at all levels does help, but it does have a large up-front cost to get it right. The key to getting stable GUI-end automation is ensuring that there's precisely one place in the script code that explicitly references the GUI components. Once you've got that, whether through name mapping ("EditBox1" = "FirstNameEdit" etc) or some other method, GUI-level automation gets a lot more stable. In the last 6 weeks I've spent something between 3 and 5 times as much time working with the oldest scripts - glorified and slightly refactored record and playback - as I have with the ones that were designed and built with around and object-oriented, data-driven script code framework.

That said, I'd guess your immediate problem is educating those around you on the value of testing and the benefits of more unit testing from all developers. That can be a sticky topic, especially when you're effectively telling the developers you work with that their work "isn't good enough". For that, I'd suggest introducing unit tests to the ones who don't unit test as a case of saving time and effort by making sure that nothing that anyone else does can break their code without everyone else knowing.

I hope that gives you a few things to think about. It's not a pleasant place to be, but sometimes knowing you're not the only one there helps a lot.

Answer 7

Very Good Question. I had gone thru similar state. Please find related question

Adapting agile testing is it good approach?

User246 answer I would like to mention and add few of my suggestions

• First, ask yourself which aspects are Agile are important to you. Second, ask yourself why those aspects are important.
• And finally, ask yourself whether the practices you mentioned are compatible with those reasons

Since, you are only one tester managing multiple applications, The challenges are

• No Scope to invest on Automation due to time constraints resulting in manual testing, This would limit test strategy to manual testing
• I would suggest you to evaluate / take time to analyse, How repetitive manual test efforts can be achieved, You need to allocate 20% of time for automation
• Probably a splitup of your efforts how it is spent. You might be working on multiple tasks resulting in working for extended hours. If you can timebox/ plan your task and verify actual efforts it would help. Try using http://www.rescuetime.com/
• If developers can execute unit test cases/ document basic functinal test cases that they execute it would help to identify bugs at earlier stage, If you can identify callout basic bugs which could have been identified by DEV team this would result in reducing time spent on fixing issues
• Adopt a mix of white-box and black box test strategy. For DB related testing, Please try tool - T.S.T - T-SQL Test Tool - http://tst.codeplex.com/, VSDB DB edition, See if you can automate Web Service, Database (Stored Procedure related test cases). This way you can manually test UI layer and other related functionality can be tested seperately. I did some analysis for a webapp - Test Automation Design Question
• Another Good Read - What factors should affect the ratio of QA/Test staff vs. App Developers

Please add your comments. I will share my feedback based on my learnings.

Answer 8

There are two points I would like to make.

First, agile does not mean so fast that you can't keep up, it is actually exactly the opposite. The goal of agile is to be fast, but with quality. One of the core ideas in the scrum model for example is to have a dedicated sprint for deployment (this is an often overlooked or ignored idea). The reason for this is to provide that stability in order to really nail down the quality. Another point that has been mentioned but is definitely worth repeating is that test driven development with ample unit testing is a core concept of agile that is very important.

Second, I would like to give a different opinion on UI automation. I have had tons of success and would consider it one of the simplest and most important tools I have available to keep on top of regression testing assuming that the life expectancy of the product you are testing is long enough and the number of regression test passes during that life expectancy is more than just a handful. In order for UI automation to be successful and maintainable, however you need to use the correct levels of abstraction. If you write your UI automation correctly then whether a code change affects one test case or a thousand test cases should not matter, the amount of work you need to do to update your tests should be the same. This means that a test case should look something like this:
public void UpdateUserTest()
{
SignIn(UserData);
UpdateUserData(UserData2);
VerifyData(UserData2);
}
This is a very basic example. Notice there are no references to specific elements on a page, or even specific pages inside of your test method. Similarly inside of your helper methods (SignIn, UpdateUserData and VerifyData) you should have only references to elements contained inside of page container classes. Lastly, developers need to be told to include ID's where appropriate to elements (and not randomly change those ID's) and be aware that this affects your automation so they can avoid that when possible.

I hope my positive experience with UI automation is useful information for some of you out there. I would be happy to provide further details on how I implement the abstractions for web UI automation if it would be useful.

Answer 9

I would like to add one point that has helped me in the past. It is about responsibilities in this kind of situation.

In my mind, it is not the tester that is responsible for creating quality. It is the developers that should create quality by writing good code. They should be able to not only believe but also show that they have created units that work. And that the units work together. The tester in this kind of situation really should only be doing a last inspection step, either accepting or failing the end result.

Now, a tester can help the developers in various ways. But, to my mind, it is important to not take away the responsibilities from the developers.

I have an example from a software product that where supposed to be distributed by IBM. The firm sent the product including documentation to the IBM test lab. The result coming back was "We have found too many errors and will not continue with more tests". When the company asked what errors the lab had found, the answer was "We will not tell you, because then you will only fix those errors and not all the other error that are bound to be there". This is of course extreme, but might serve as an illustration.