6

In an ideal world we'd have Acceptance Tests-->Unit Tests-->Integration Tests-->Test Plans and Automated GUI Tests (With Selenium or something).

But where does it start to get cut out when you just don't have the testing resources?

Unit Tests are usually created by devs, so thats less of a problem.

But Test Plans/Integration Tests (With Rspec/Cucumber/etc...) and then writing Plain ol Test Plans (Old school style: Expected Result/Actual Result/Steps/etc...)

Im the Sole QA, so with multiple projects it's basically impossible for me to do so, the company is rather small...so hiring more testers/QA isn't always the option. It seems to me that having Acceptance Tests is a must have, and possibly testing sessions for bugs...but I'm wondering if the old school type of test plan....really make sense?

To me, they seem lengthy and time consuming, and also take away any possible "Automation" R&D and implementation, which is what I'm really struggling with. However Automation and implementing some sort of CI process also takes a lot of time, which could be used for writing test plans.

Right now i've been researching Automation testing in my spare time trying to figure out the best course of action, while doing exploratory sessions and recording the bugs in Excel (Lame I know, but I gotta start somewhere)....Where do you go from here?

7

The short version: Don't sweat on automation or test cases. Chances are you won't get time to build it because there's too much that has to be manually tested now. Instead focus on building light-weight, easy to maintain documentation of the key elements of the software you're testing.

The longer version:

This is my world. I'm the sole tester in a team of 10, supporting multiple applications and projects. To make things even more entertaining, the applications we support are not unit tested (mostly because it's next to impossible - classic ASP does not allow unit testing, and old-school VB6 may be worse).

The suggestions below are based on my experience:

  • Start by defining a short list of tests to use as a smoke test (in my case "short" means it takes about an hour to run these manually). These should cover the most critical functionality of the software and/or the things users do most. It doesn't matter how you define them, as long as you can easily reference them. These are the tests you run for every release of the application in question.
  • Keep your test cases for enhancements and new functionality at a high level. I typically give a short descriptive scenario for the test case, such as "User with admin profile can see the button. On click, the button takes the user to site X". In my experience you only need more detail if you've got a developer who codes exactly what's in the specs without any thought to wider implications (and by "exactly" I mean if the design doc doesn't explicitly say how many pixels spacing between page elements the page elements will have no spacing - and then argues that the code matches spec so it's not a bug).
  • If you can use an integrated application lifecycle management tool, do it. If not, use whatever works. I've found wikis are useful if there's no ALM available. (The advantages of ALM tools like Team Foundation Server and so forth is that your builds, test cases, source control, and bug/issue management tools are all in one central repository which makes cross-linking a lot simpler)
  • Domain knowledge is your friend. Talk to as many people as you can about modules that tend to have problems, external dependencies, usage patterns, customers you can't afford to upset, and so forth. This helps you work out the highest risk areas.
  • Document dependencies. You don't need much detail here: I created a set of wiki pages (using an antique version of SharePoint's built-in wiki engine) that start with a list of all the modules in the application and which ones can be affected by changes to other modules. Each module has a page describing (briefly) the page function, the conditions where the module is read-only, and a bulleted list of things I need to consider any time I have to test changes to that module.
  • You will become the go-to person for application knowledge. This is why I suggest if you have to look up something more than once, you put a reference to it somewhere easily located. Software developers typically develop in-depth knowledge of the modules they work with. Testers and QA people typically develop a broad knowledge of the software, and when you're the one and only, that means you wind up knowing how to do almost everything the application allows you to do (and usually quite a few things it shouldn't let you do).
  • This is all web-based stuff unfortunately so...there isn't much in the way of "learning" the software. It's all websites and a few mobile apps and each one is completely different. Although I do agree with what your saying, I'm just trying to find some middle ground lol. As far as the ALM we use Jira, but I feel like keeping test cases in Jira is clutter some...what do you use if you don't mind me asking? – Mercfh Jul 14 '15 at 13:37
  • I use Rally - which I find rather unfriendly for test case management but that's what the company requires so that's what I do. I don't use a fraction of the detail information that could be entered for test cases - usually not much more than a title and a description. – Kate Paulk Jul 15 '15 at 15:37
  • Yeah i've used rally before, and quite honestly I didn't like it. Thankfully it's kinda up to me what we use "QA" wise, besides Jira. Good to note about ur test cases...it seems super detailed ones (from what I can tell) are a waste of time for most people. – Mercfh Jul 15 '15 at 15:44
  • 2
    for test case management I've been using TestRail, I love it, It still lacks some functionality, but overall this is by far the best test management software I've used (Jira/HP/Rally are really not meant for it). It's like a user friendly testlink – StanM Jul 27 '15 at 14:17
2

Ouch. No, the old school sort of test plan doesn't make much sense in your context. The only reason I can see for doing them is if you had a contractual requirement with a client to deliver them.

Will anyone read them except you? (If no - then stop doing them now!)

If you re-read them, what do you use them for?

To remind you of how to do something (is it already in user docs? Is there a team wiki where you can document how to set up some tricky config once, instead of in multiple places? Or better still, get other people to do it as well!).

Or to remind you what you did last time (can you just use one-liner test ideas, or mindmaps to help you note down details in a very fast and frugal way?)

The other thing that you may be able to do (depending on politics) is to step back from trying to do all the testing, and instead advise on all the testing. I.e. pair with developers/product owners to discuss/plan what needs to be covered, and then help them to do this, rather than trying to do it all yourself. This is the most sane approach (using a scarce resource as a coach rather than trying to spread them so thin they can't get anything done), but it's also considered utterly unimaginable crazy fool stuff in too many organisations. I don't know what your context is. You may find that you can do a little bit of this - e.g. get a friendly developer on one project to help you implement automated acceptance checks, but other devs may not be willing.

  • 1
    Your first comment (will I read them) is exactly why I don't think it makes sense to do them. I guess I'm trying to fill in the gaps...IE if I don't do that then what do I do (besides ofc attempting to automate). I initially thought about using cucumber, and at least making feature files for all the user stories as acceptance tests...that way even if I don't have time to automate them...they are still there. On your last point however I don't think that will work, since they def. want me to help bring up the QA dept. including test. Tough stuff lol – Mercfh Jul 14 '15 at 13:36
2

You mentioned having a handle on testing, but wanting to know more about documentation in limited time scopes.

In that case, my advice would be to get into Jira or another bug-tracking software asap. I like to keep a regression testing google doc spreadsheet which I then insert links to relevant bugs in Jira. That way you, fellow testers/devs/PMs, can all see previous bugs and get an idea of which areas are vulnerable over time.

Then for each release, you use the same doc, just adding a new column. That way, you have everything in one place, rather than multiple excel files that are difficult to go through.

Does that make sense? I can describe in more detail if you like or answer any questions.

  • Im confused at what you mean by adding a new column? What is the new column exactly? – Mercfh Jul 30 '15 at 14:14
1

Both Acceptance tests and Integration tests could be either manual or automated. Unit tests make sense ONLY automated, and I assume your developers write unit tests (as they should).

Start with manual testing most often used and most recently changed functionality first, where detecting a problems has best value for time invested. But ask your manager for permission to spend some time on automation (instead of manual testing of some obscure functionality), to automate regression testing.

Automate most often used functionality first, and add more as you have time, possibly saving time by letting program to test basic functionality.

1

It seems to me that having Acceptance Tests is a must have, and possibly testing sessions for bugs...but I'm wondering if the old school type of test plan....really make sense?

Acceptance tests are based on a ticket's Acceptance Criteria. A testing session for bugs... I think you mean a smoke test. Which I advice before every deployment, at a minimum. The "old school test plan" is exhaustive and benefits no one. Check out our discussion on this topic.

Right now i've been researching Automation testing in my spare time trying to figure out the best course of action, while doing exploratory sessions and recording the bugs in Excel (Lame I know, but I gotta start somewhere)....Where do you go from here?

If you want to learn automation, most people start out with Selenium IDE. Get your bugs into a shared system. At least, use Google Docs so you can share the sheet easily with devs and no one who wants to look at it will need M$ Office. There are plenty of online tools for Issue Tracking. A popular one is JIRA. I have used alternatives like BugTracker and FogBugz, but JIRA is by far my favorite (beating out TFS, too).

  • Right now I'm actually using sheets to track bugs before putting them into Jira. I think I'm just trying to figure our what to do with test plans in general. But yeah automation may be something I just have to dive into. But figuring out what to automate is gonna be difficult with just me (IE what will I have TIME to automate lol) – Mercfh Jul 14 '15 at 19:57
  • I site Benchmark QA's "Top 25 Tips for Better Software Quality" that is in agreeance with "The Art of Software Testing" in that you only automate "what is worth it". Finding that balance is hard and will take some trial and error. I can tell you that I only automate admin stuff and MVP functionality. Everything else is less critical. You could automate everything, but I can tell you in an understaffed or agency setting that is more work than is reasonable. – kirbycope Jul 14 '15 at 20:05
  • Seems that way, especially me being a 1 man show...but hey gotta start somewhere. I wouldn't mind maybe making the acceptance test cases in Cucumber at least (or something similar), that way if we decide to automate them for regression tests later...the layout would already be there? – Mercfh Jul 15 '15 at 14:03
  • I would just say, "remember your audience". If only you are looking at Test Plans, do whatever works for you. However, if you might be asked to share it with a client then you'll want to write a test plan with technical people in mind. – kirbycope Jul 15 '15 at 18:36
  • Thats why I think a Cucumber style Acceptance Test(s) might work the best...since A. Future Automation Possibilities, B: It's sorta constrained to a certain "format" and C: Devs and Non-technicals can understand it. – Mercfh Jul 15 '15 at 18:41
1

The first three things that come to my mind are:

  • Can you get the product owner to test new functionality? As they know what they want (as opposed to what they asked for)
  • Don't have a test plan, have a one page check list that you can print off and tick off, just so you don't forget any major areas of functionality.
  • Start automating simple things, and think off them as regression tests. It's easy to break rarely used (by developers/testers) and harder to test functionality that is important to end users e.g forgot password
1

Kate is dead-on with her suggestion about the short version.

If you are REALLY pressed for time, these are the steps I take:

You want to save your team time and pain, and you do that by focusing on potential issues that are going to be much less costly if we catch them sooner rather than later. I usually start by asking myself "what feature(s), if broken will…”

  • block me from the majority testing,
  • be particularly difficult/time-consuming for the developer, or
  • require fixes that are potentially wide-reaching and will therefore result in me needing to re-test other scenarios in my test plan.

The faster I get those bugs back to my developer, the faster he or she can commit a fix. The earlier in the process I do that, the less likely it will be that I’ll need to re-test things that the new commit might have impacted.

  1. Test the most basic happy path surrounding NEW code, for 10 minutes. If you aren't able to perform the most basic functions of a new feature, you need to get the code back in the hands of your developer as soon as possible. Chances are, if something is broken here, it could be a longer fix.
  2. Test functionality that impacts a large percentage of the code base as a whole. This is where we find dependencies that we didn't think could possibly be impacted by our changes. It’s much better to find these early, because they could indicate a need for deeper code reviews, and also wider regression testing and an adjustment to your test plan.
  3. Test the critical functionality, and the most commonly used areas. What does your app NEED to be able to do? What do most people use it for? What are the potential blockers that are the only reason some users use your site or app?

  4. Test the potential areas of high risk. Data loss. Load/Performance issues. This should be part of a conversation you have as a team - what are the biggest risks involved in this sprint? What are the worst-case scenarios? This is also the time where it’s most beneficial to test in a stage environment that replicates production as closely as possible.

  5. Then normal regression testing on down to edge cases and thinking of creative ways to break stuff (my favorite part).

Depending on how limited time is, you will have to make tough decisions of what to shorten and cut based on the situation.

Lastly, when you can do documentation that takes a short amount of time now, but will save exponential amounts in the future - DO IT. Make the time, even if it's not until later after the release goes out. You will then always be making the most of limited time.

  • All good advice thanks! I think the actual "testing" portion my job I feel confident at, it's just actual documentation for the team (IE test plans etc..). Im trying to find something that's "doable" by just me in the allotted time. – Mercfh Jul 20 '15 at 13:54
1

I'd like to add two more points which will be difficult for firms but as this topic is not limited to this (at least in my opinion) two other possibilities. Both can't be done by QA only.

1) Make your code open source. By making the code open source you'll get more reviewers and more feedback on source code and there will be more installations of your product with more practical users.

2) A thing that mostly games could do, but also other software, Inspired by Blizzard: Open a public test realm where players can play preview versions and give feedback about issues. For "normal" software a open demo version (like testlink eg) could be a similar way to go. This is something the game for which I code in my freetime does this now as we don't have any "internal" testers.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.