7

I recently moved into the Software Testing department of my organization.The business has little history of automating testing, and most software development and testing is conducted by third parties on our behalf. However, we do employ testers directly, and are now looking to increase automation within software testing.

To do this, we have begun to purchase test scripts for regression packages from these third parties. I have a bit of programming experience, and I worry that purchasing scripts will not allow us to have a) the knowledge to evaluate these scripts effectively and b) to fully access the benefits of automation.

By the full benefits of automation, I mean the capacity to understand on the spot whether or not a process is a potential candidate for automation, how long it would take to automate versus manual execution, how to effectively re-use existing function libraries (test partners and the individuals working for them regularly change) etc. This knowledge could lead us beyond simply automating regression packs for long term maintenance purposes, to more sophisticated automation approaches.

However, as no such capacity exists within the organization at the moment, there would be costs associated with creating and maintaining the skill base required.

As someone new to testing, I wonder - are my concerns justified, or are the cost savings associated with buying test scripts sufficient justification for a lack of in-house expertise?

4 Answers 4

7

You're going to get a lot of "it depends" answers for this. Whether it's better to stay with primarily third-party automation will change depending on the quality of your third party automation providers, the nature of your business applications, your internal infrastructure, the method your third party people use and a whole lot more.

Some of the factors to consider are:

  • Does your business build small, self-contained applications or maintain a large, complex application suite that gets new features added to it? In the former case, third parties could be the better option, where a continually changing complex application requiring a lot of domain knowledge is a better candidate for internally developed and maintained automation.
  • What is the automation/coding skill base of your internal test team? If you have a number of testers with good formal logic skills and a decent programming knowledge, you have enough to build good, maintainable internal automation. If your test team is primarily manual without much in the way of formal logic, it's going to be a lot more difficult to do this and you'll probably hit resistance from your test team.
  • What are your organization's budget priorities? Typically a third party automation solution is going to automate existing manual test cases, something that can be done fairly quickly using the third party's pre-existing structures and frameworks. Building your own from scratch (because chances are there's a lot of proprietary code in the third party automation) is very expensive in terms of resources, time, and money. The third party automation could use a tool that is outside your budget, giving you no ability to reuse their material.
  • What are your organization's long term plans? Your test strategy - including what gets automated and how - needs to support the organization's long range goals. Using a third party to build and maintain automation may be more aligned to the long range goals than building your own.

Some of the advantages of building your own automation are:

  • the automation is built by people with domain knowledge.
  • if it's done right, adding new tests to existing automation is a lot easier.
  • your organization retains full control over the automation.
  • your team is more familiar with the automation, making it easier to ensure that any test failures are real problems with the software.

Some of the disadvantages are:

  • it can take a long time to build up a stable, reliable automation suite.
  • depending on the application technologies you need, automation tools can be extremely expensive.
  • you need testers who can code to at least intermediate developer level. Ideally you want people who can fill the Software Developer in Test role, but those aren't all that easy to come by (the combination of good test skills and good coding skills isn't terribly common).
  • you will need to convince management of the need and then keep them convinced.
  • from the management perspective, it's easier to contract someone to build automation, particularly when that's the way things already work and the third party is experienced with building automation. It may also be cheaper.
  • it's extremely difficult to quantify potential cost savings from moving automation in-house, especially when the only thing management is likely to see is increased expense.
2
  • 1
    Thanks, that's really clarified the pros and cons. In fact, the business is quite long-lived, and has precisely the huge, complex, hybrid architecture (with many custom application interfaces) you describe, which it has evolved over an extended period of time. There are no plans to change this. However, the appetite for the costs involved in developing programming capacity (which we currently lack) may be difficult to create. I think both the answers here provide a good basis for further research though, so thank you. Apr 26, 2014 at 8:31
  • @user3100456 - I quite understand. At my last position, the core application had been evolving for 25 years (and still had some of the original code). The automation had been around for 10 years and was just getting mature enough to be reasonably easy to add new tests to. Most of the customer-reported bugs were edge cases or in gaps in the automation, but it was still a constant struggle to keep management invested in the automation effort.
    – Kate Paulk
    Apr 30, 2014 at 10:58
6

A lot of this will depend on if you are getting a good return of investment from your third part testers and testing scripts. If the end result if a good regression suite with the ability to vary inputs and conditions at a reasonable price then there is less urgency to take the automation fully in house.

Even if the third part testing scripts provide a good return of investment there is likely to be limitations. If the test scripts break then you have to wait for the third party to fix them, if the behaviour of the produced software is to change then you have to communicate these changes to the third parties which can be a challenge. And there are the concerns you mentionded,: Is every test that could be automated actually automated? Do we know what is worth automating and what isn't? Are the automated tests well designed and make effective use of existing libraries? These are questions and limitations that you can solve if you take control of the test automation.

You mentioned that "that there is little history of test automation in the business" and that you've only "begun to purchase test scripts" which sounds like you are starting from scratch and there is no previous automated test set up. To me this is the perfect time to take the test automation in house. My advice would be take your time and build up your own test automation scripts whilst still making use of your third party scripts, once your own scripts are good enough then you can look to end the third party scripts subscriptions.

5

All excellent answers. I will add one more thought. Several years ago I was responsible for a team in China that did a lot of software internationalization testing on the products of IBM's Tivoli unit. These were huge and complicated products and since the team really had to exercise more-or-less every menu choice and verify that the displayed results were reasonable, these tests amounted to a pretty extensive system test.

At any rate, that team had a lot of very bright engineers on it and they did a number of experiments with automation. Their conclusion was that the best ROI was not from trying to automate the test itself (complex interactions, constantly changing features, all of the problems mentioned above) but rather to automate the setup of the test lab. That is, when they looked at where the effort went, setting up the test environment (blanking harddisks, reinstalling OSs, etc... for perhaps 100 machines) was a surprisingly large part of the effort and was quite tractable to automation. They went ahead and took did so, got a lot of drudge work automated, and just did the testing itself manually.

Of course that result was very specific to the type of product and type of testing they were doing. However, before you jump feet first into automation (especially purchasing external products and services) you might want to do some careful time and effort tracking first to see what is really eating up the time. You might be surprised.

2
  • Fascinating answer. You're right, close analysis of time/effort actually invested in the overall process could open up alternative targets for automation. Once again the skill base is something of an issue - you need "bright engineers" who are able to analyze the overall objectives in the context of their own sound automation skill, and are then able to direct that knowledge to creating less conventional solutions. Apr 30, 2014 at 18:16
  • omg +1 +1000 if I could. THIS was the real problem at my last employer for 18 months. The term "sorry, I've got configuration issues" became a dreaded, but oft heard phrase. Towards the end I started writing a lot of shell scripts to automate build and setup procedures. In the current trend of multi-service architecture with multiple repositories and services that need to be started, this makes a big difference Jun 18, 2015 at 10:29
3

Both Badlands and KatePaulk gave great answers. I would simply add that for a complex application/architecture like your company's, in-house testing automation has a lot of benefits that you simply cannot get from off-the-shelf testing scripts. In my experience, it is when I sit down and think, "How will I test (or automate) this feature (or test)?", that I discover architectural problems and even outright bugs. In other words, sometimes it's not merely the testing that matters, but the analysis of how to test something accurately, repeatably, and efficiently. When you outsource that, you aren't forced to do the same analysis, i.e., "Why does this test take forever and use tons of resources?"

Aside from that, a one-size-fits-all testing script/suite can only go so deep. It can't hit that legacy code that is horribly full of anti-patterns, un-SOLID, non-DRY kludge, but that your organization "hasn't gotten around to fixing" because it's 20,000 lines of nightmare-waiting-to-happen. On the other hand, once you and your team understand it well enough to test it thoroughly and automate your tests, you understand it well enough to refactor it quickly (and accurately, thanks to automatic testing as you incrementally improve the code).

In short, developing an in-house testing and automation skill base will pay huge dividends and allow you to improve your application and architecture much more quickly.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.