5

i've just started at a Web Development company that does websites/mobile apps. I've done QA for 5-6 years now and they wanted someone that had QA experience but wasn't set in their ways to be their QA lead for their new QA dept. (Which just consists of me).

In the past the company had kinda taken an ad-hoc approach, no real standardized approach to collecting requirements etc... However now they are doing a normal Scrum approach, tracking user stories in Jira (Which I think personally is a good start).

In my previous job we had very strict requirements defined at the beginning of a project (Like a LOT of requirements) based off an FSD (Functional Spec Document). I would write my test cases based off these.

However User Stories tend to be a little more generic and vague while still identifying the requirement.

Whats the best process for really creating test cases/test plan based off of this. I feel like conventional test cases (IE: User clicks this button: Steps: Expected Result: etc...) seems a little...slow and cumbersome? and requires a lot of maintenance especially if things change?

Obviously automating everything is impossible...but is there like a halfway point maybe? My idea is to create acceptance tests based off the user story, I've read about cucumber and Gherkin...which seems cool because you can write a High level Acceptance test that can be automated later. This kinda covers the Front Acceptance Test end while making automated tests later not be such a hassle.

Am I missing anything? I guess I wonder about all the "conventional" test cases...and if/when they are needed. After my experience with doing them and spending more hours than i'd like to admit writing STEPS/EXPECTED OUTCOME/Etc.... x10000 they just don't seem as needed...is this wrong? I just wanna make sure what I said doesn't sound....well stupid.

2

To start with, I'd suggest browsing the answers of the questions showing as related, particularly Test Automation in Agile? and Agile Tester vs Traditional Tester...........?

As far as test case writing, it doesn't really matter how you handle that, as long as someone who is moderately familiar with the application can follow your test cases. The main things you need are the goal of the test case (I often state this at a high level in the title of the test case, e.g. "Users with permission level admin or higher can access feature X"), the desired outcome (you can take "does not crash" as implicit in all of these if you have a stability non-functional requirement as part of the user stories), and the actual results. That's the minimum necessary, and can be stated in English or in a Cucumber-style fixture. I'll typically state the generic case, then have a list of data-specific cases I plan to look at (usually exploring around boundaries and crucial configurations).

The approach you take is going to depend on the dynamics of your team and the nature of your web applications, but I can make some suggestions here:

  • Developers should be writing unit tests and these should form the bulk of the automation. Your role with these tests is to work with developers to generate good unit tests.
  • Developers should be writing module/integration tests and these should be your next largest number of tests. Once again, you need to work with developers to help generate good tests.
  • Testers or Developers write system-level tests. These tests have some overlap with integration tests, but largely cover system processes from end to end. If you can't do this kind of testing through your application API you have a problem.
  • All the unit, integration, module and system tests should be automated and run with every build. These are all "headless" tests in that they don't interact with the GUI, so they run quickly.
  • If there is any way to avoid GUI automation short of manually testing everything, use it. GUI automation is slow, expensive and even with the best design can require a large maintenance cost. You want to avoid this wherever possible.
  • Tester-built automation should happen early in the next sprint. The reason for this is that automating something that hasn't stabilized is wasted time. The ROI for any automation lies in future releases, not in the current cycle.
  • When starting, look for high value and easy to automate and maintain tests. This is particularly important when you're looking at something with several years of development and no automation of any kind in place.

Also - and this is particularly important - You may not be able to automate everything. Many of the most experienced folk here will tell you that you should automate 100% of the tests that should be automated - the fun is in working out what those are.

Some things that can't or shouldn't be automated are:

  • Look and feel - until computers can beat the human eyeball and brain at pattern recognition and complain about awkward/unwieldy processes, this has to be done manually.
  • Not all widgets (web or otherwise) can be accessed by automation tools.
  • You can automated checking printouts, but it's clumsy, time-consuming, and very error-prone. The alternative: automate checking the printout text and run a printout for a manual lookover is a whole lot easier, faster (and therefore cheaper) and less likely to break.
  • Unless you know how power-users actually use your application you can't automate that. When you do know how they use it, the best you're likely to be able to do is make sure all the shortcuts keep working.
  • I guess with the Unit Tests/Integration I'm not sure how to do that on a web platform, I just started so I'm still getting use to everything. Since were a web dev company (mainly rails). I think automating the acceptance tests and where/how is where I'm really struggling – Mercfh Jul 10 '15 at 18:26
  • @Mercfh, you might want to work with your developers for this: if they have a way you can run the acceptance tests headless, you can write them in a Cucumber-like language and they can help/teach you to code the details. You'll still need to decide what should be automated, but it will be a starting point. – Kate Paulk Jul 13 '15 at 11:42
  • Also do you mind me asking what sort of software you use to write your test cases? and how long the typically are? And are you just talking about acceptance test cases or generic ones – Mercfh Jul 30 '15 at 14:22
  • 1
    @Mercfh - I'm currently using Rally, which isn't optimal. I've used MS Test Manager, TestLink, Word, and Excel in the past. I'm discussing all types of higher level tests: acceptance tests, functional tests, and so forth. I generally try to keep them short and cover the essentials - press this button, click this link style tends to get out of date fast, is boring for manual testers, and takes far too long to write compared to the benefit received. – Kate Paulk Jul 31 '15 at 13:06
  • Yeah I'm trying to avoid that as well, these test charters i've been experimenting with have been helpful in that regard for sure. – Mercfh Jul 31 '15 at 13:46
2

You are on the right track. I used to do the granular test plans until relatively recently. I have gone the direction of making a Test Plan mostly consisting of User Stories. You need to remember your audience when writing anything. My Test Plans are part of the deliverable to clients so it makes the most sense that anyone can read it and use it. That means cutting out the granular, technical stuff that made something you could explain in 5 sentences into a 200 row spreadsheet.

The part about automation shouldn't be a major factor this early into the game. As J M Juran says, people want to automate to be quicker but they end up automating bad processes. I recommend you take the time to iron out your manual processes and then decide what is worth automating and how.

  • Sorry but I have to downvote. I think giving advice to only automate what is "worth" it is not a good practice. You have to think upfront how to automate everything if you want to scale in the new modern continuous delivery world. How will your company compete with companies that release new features daily... – Niels van Reijmersdal Jul 10 '15 at 7:25
  • 1
    I dont need to compete with those companies. Automating everything is not a good idea, especially in my agency setting. Automating what is worth it is a best practice from Benchmark QAs 25 Tips and also the book The Art of Software Testing. If you want to automate everything, have fun. – kirbycope Jul 10 '15 at 12:36
2

I had the exact same questions as you when I started the "testing" compartment in my current Agile project.

It is indeed not feasible to write and maintain test cases in the tradition format. User stories are delivered every few weeks, and change rapidly afterwards. Test cases would become obsolete faster than you can write them.

One approach that I found useful in our small Agile team is session based testing. You can find some explanation here and here.

Because we're using Microsoft Test Manager, I've created a custom template based on this SBT method. (No use sharing it here because it's not in English.)

The main advantages are: one test charter is linked to all related bugs and test execution history. One test charter also contains all relevant tips and tricks needed to test the user story in question. There's little administrative overhead. Worth looking into, as you can easily adapt this idea to your needs.

  • Sounds Similar to what I'm trying to accomplish, since the format I was looking at what gherkin (and maybe integrating it into cucumber) – Mercfh Jul 10 '15 at 18:24
  • Side-Question: Do you write any actual test plans based on the User Stories? also what software do you use besides Microsoft test Manager? – Mercfh Jul 13 '15 at 13:12
  • 1
    Our "test plan" (in MTM terminology) simply consists of the test charters for the user stories of a given sprint. Each charter has the main goal, what's needed for set-up, references (like the user story, SQL queries, etc.) and activities (specific tests, tips and tricks, ...). Also we don't bother with desired outcome versus actual outcome on test case level, for reasons stated before. So our testers need to have decent knowledge of the system in order to test well. – FDM Jul 13 '15 at 14:40
  • Hmmm not sure something like that would work as well for us, I'm just thinking out loud but since were Web-Dev there isn't much in the way of setup really for us, at least not anything that complex that differs that much each time. I can def. see the use in a charter however. – Mercfh Jul 13 '15 at 14:45
1

Obviously automating everything is impossible...

How do you think teams that practice continuous delivery are working?

Being Agile means that you optimize the feedback cycle on all levels, for long running projects a rule of thumb should be to automate everything. This means the whole pipeline up-to deployment in production.

I would rather say: of-course automating everything is possible!

Just work with the development team and make it testable. Cut out any pieces of infrastructure or external libraries that are "untestable" and replace them with something else that is testable.

I love the BDD concepts, but if your team is not using it as communication tool between with the business it might be a layer of waste, since it adds overhead when creating automated tests.

you can write a High level Acceptance test that can be automated later.

The automation should be done before releasing the functionality described in this acceptance test. If you don't you will start falling behind with test automation. How can the team release new features that build on top of these in the next iteration? Manual testing? Writing and executing manual test-cases is not sustainable in a ever fast changing world.

  • Automating everything isn't possible, but you can get close depending on the application and the requirements. More in a separate answer. – Kate Paulk Jul 10 '15 at 12:54
  • I think since the team is new to the whole scrum process, Continous delivery/integration is the goal...but it'll be some work. Just trying to figure out things I can do for now at least. And I'm not sure where to start with automating acceptance tests (since dev automate unit tests) – Mercfh Jul 10 '15 at 18:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.