What to do with Limited Time/Testers - What is most Important?

Question

In an ideal world we'd have Acceptance Tests-->Unit Tests-->Integration Tests-->Test Plans and Automated GUI Tests (With Selenium or something).

But where does it start to get cut out when you just don't have the testing resources?

Unit Tests are usually created by devs, so thats less of a problem.

But Test Plans/Integration Tests (With Rspec/Cucumber/etc...) and then writing Plain ol Test Plans (Old school style: Expected Result/Actual Result/Steps/etc...)

Im the Sole QA, so with multiple projects it's basically impossible for me to do so, the company is rather small...so hiring more testers/QA isn't always the option. It seems to me that having Acceptance Tests is a must have, and possibly testing sessions for bugs...but I'm wondering if the old school type of test plan....really make sense?

To me, they seem lengthy and time consuming, and also take away any possible "Automation" R&D and implementation, which is what I'm really struggling with. However Automation and implementing some sort of CI process also takes a lot of time, which could be used for writing test plans.

Right now i've been researching Automation testing in my spare time trying to figure out the best course of action, while doing exploratory sessions and recording the bugs in Excel (Lame I know, but I gotta start somewhere)....Where do you go from here?

Answer 1

The short version: Don't sweat on automation or test cases. Chances are you won't get time to build it because there's too much that has to be manually tested now. Instead focus on building light-weight, easy to maintain documentation of the key elements of the software you're testing.

The longer version:

This is my world. I'm the sole tester in a team of 10, supporting multiple applications and projects. To make things even more entertaining, the applications we support are not unit tested (mostly because it's next to impossible - classic ASP does not allow unit testing, and old-school VB6 may be worse).

The suggestions below are based on my experience:

• Start by defining a short list of tests to use as a smoke test (in my case "short" means it takes about an hour to run these manually). These should cover the most critical functionality of the software and/or the things users do most. It doesn't matter how you define them, as long as you can easily reference them. These are the tests you run for every release of the application in question.
• Keep your test cases for enhancements and new functionality at a high level. I typically give a short descriptive scenario for the test case, such as "User with admin profile can see the button. On click, the button takes the user to site X". In my experience you only need more detail if you've got a developer who codes exactly what's in the specs without any thought to wider implications (and by "exactly" I mean if the design doc doesn't explicitly say how many pixels spacing between page elements the page elements will have no spacing - and then argues that the code matches spec so it's not a bug).
• If you can use an integrated application lifecycle management tool, do it. If not, use whatever works. I've found wikis are useful if there's no ALM available. (The advantages of ALM tools like Team Foundation Server and so forth is that your builds, test cases, source control, and bug/issue management tools are all in one central repository which makes cross-linking a lot simpler)
• Domain knowledge is your friend. Talk to as many people as you can about modules that tend to have problems, external dependencies, usage patterns, customers you can't afford to upset, and so forth. This helps you work out the highest risk areas.
• Document dependencies. You don't need much detail here: I created a set of wiki pages (using an antique version of SharePoint's built-in wiki engine) that start with a list of all the modules in the application and which ones can be affected by changes to other modules. Each module has a page describing (briefly) the page function, the conditions where the module is read-only, and a bulleted list of things I need to consider any time I have to test changes to that module.
• You will become the go-to person for application knowledge. This is why I suggest if you have to look up something more than once, you put a reference to it somewhere easily located. Software developers typically develop in-depth knowledge of the modules they work with. Testers and QA people typically develop a broad knowledge of the software, and when you're the one and only, that means you wind up knowing how to do almost everything the application allows you to do (and usually quite a few things it shouldn't let you do).

Answer 2

Ouch. No, the old school sort of test plan doesn't make much sense in your context. The only reason I can see for doing them is if you had a contractual requirement with a client to deliver them.

Will anyone read them except you? (If no - then stop doing them now!)

If you re-read them, what do you use them for?

To remind you of how to do something (is it already in user docs? Is there a team wiki where you can document how to set up some tricky config once, instead of in multiple places? Or better still, get other people to do it as well!).

Or to remind you what you did last time (can you just use one-liner test ideas, or mindmaps to help you note down details in a very fast and frugal way?)

The other thing that you may be able to do (depending on politics) is to step back from trying to do all the testing, and instead advise on all the testing. I.e. pair with developers/product owners to discuss/plan what needs to be covered, and then help them to do this, rather than trying to do it all yourself. This is the most sane approach (using a scarce resource as a coach rather than trying to spread them so thin they can't get anything done), but it's also considered utterly unimaginable crazy fool stuff in too many organisations. I don't know what your context is. You may find that you can do a little bit of this - e.g. get a friendly developer on one project to help you implement automated acceptance checks, but other devs may not be willing.

Answer 3

You mentioned having a handle on testing, but wanting to know more about documentation in limited time scopes.

In that case, my advice would be to get into Jira or another bug-tracking software asap. I like to keep a regression testing google doc spreadsheet which I then insert links to relevant bugs in Jira. That way you, fellow testers/devs/PMs, can all see previous bugs and get an idea of which areas are vulnerable over time.

Then for each release, you use the same doc, just adding a new column. That way, you have everything in one place, rather than multiple excel files that are difficult to go through.

Does that make sense? I can describe in more detail if you like or answer any questions.

Answer 4

Both Acceptance tests and Integration tests could be either manual or automated. Unit tests make sense ONLY automated, and I assume your developers write unit tests (as they should).

Start with manual testing most often used and most recently changed functionality first, where detecting a problems has best value for time invested. But ask your manager for permission to spend some time on automation (instead of manual testing of some obscure functionality), to automate regression testing.

Automate most often used functionality first, and add more as you have time, possibly saving time by letting program to test basic functionality.

Answer 5

It seems to me that having Acceptance Tests is a must have, and possibly testing sessions for bugs...but I'm wondering if the old school type of test plan....really make sense?

Acceptance tests are based on a ticket's Acceptance Criteria. A testing session for bugs... I think you mean a smoke test. Which I advice before every deployment, at a minimum. The "old school test plan" is exhaustive and benefits no one. Check out our discussion on this topic.

Right now i've been researching Automation testing in my spare time trying to figure out the best course of action, while doing exploratory sessions and recording the bugs in Excel (Lame I know, but I gotta start somewhere)....Where do you go from here?

If you want to learn automation, most people start out with Selenium IDE. Get your bugs into a shared system. At least, use Google Docs so you can share the sheet easily with devs and no one who wants to look at it will need M$ Office. There are plenty of online tools for Issue Tracking. A popular one is JIRA. I have used alternatives like BugTracker and FogBugz, but JIRA is by far my favorite (beating out TFS, too).

Answer 6

The first three things that come to my mind are:

• Can you get the product owner to test new functionality? As they know what they want (as opposed to what they asked for)
• Don't have a test plan, have a one page check list that you can print off and tick off, just so you don't forget any major areas of functionality.
• Start automating simple things, and think off them as regression tests. It's easy to break rarely used (by developers/testers) and harder to test functionality that is important to end users e.g forgot password

Answer 7

Kate is dead-on with her suggestion about the short version.

If you are REALLY pressed for time, these are the steps I take:

You want to save your team time and pain, and you do that by focusing on potential issues that are going to be much less costly if we catch them sooner rather than later. I usually start by asking myself "what feature(s), if broken will…”

• block me from the majority testing,
• be particularly difficult/time-consuming for the developer, or
• require fixes that are potentially wide-reaching and will therefore result in me needing to re-test other scenarios in my test plan.

The faster I get those bugs back to my developer, the faster he or she can commit a fix. The earlier in the process I do that, the less likely it will be that I’ll need to re-test things that the new commit might have impacted.

1. Test the most basic happy path surrounding NEW code, for 10 minutes. If you aren't able to perform the most basic functions of a new feature, you need to get the code back in the hands of your developer as soon as possible. Chances are, if something is broken here, it could be a longer fix.
2. Test functionality that impacts a large percentage of the code base as a whole. This is where we find dependencies that we didn't think could possibly be impacted by our changes. It’s much better to find these early, because they could indicate a need for deeper code reviews, and also wider regression testing and an adjustment to your test plan.

3. Test the critical functionality, and the most commonly used areas. What does your app NEED to be able to do? What do most people use it for? What are the potential blockers that are the only reason some users use your site or app?

4. Test the potential areas of high risk. Data loss. Load/Performance issues. This should be part of a conversation you have as a team - what are the biggest risks involved in this sprint? What are the worst-case scenarios? This is also the time where it’s most beneficial to test in a stage environment that replicates production as closely as possible.

5. Then normal regression testing on down to edge cases and thinking of creative ways to break stuff (my favorite part).

Depending on how limited time is, you will have to make tough decisions of what to shorten and cut based on the situation.

Lastly, when you can do documentation that takes a short amount of time now, but will save exponential amounts in the future - DO IT. Make the time, even if it's not until later after the release goes out. You will then always be making the most of limited time.

Answer 8

I'd like to add two more points which will be difficult for firms but as this topic is not limited to this (at least in my opinion) two other possibilities. Both can't be done by QA only.

1) Make your code open source. By making the code open source you'll get more reviewers and more feedback on source code and there will be more installations of your product with more practical users.

2) A thing that mostly games could do, but also other software, Inspired by Blizzard: Open a public test realm where players can play preview versions and give feedback about issues. For "normal" software a open demo version (like testlink eg) could be a similar way to go. This is something the game for which I code in my freetime does this now as we don't have any "internal" testers.