Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I've inherited a collection of desktop applications that currently have (for all intents) no automated test coverage. Generally speaking, these applications are typically deployed as a suite, installed on different machines, and communicate via a complex, undocumented, and proprietary protocol. They also communicate with vendors via socket or serial to send/receive data.

I want to introduce automated integration testing for these applications, but haven't found many tools that advertise this feature. TestComplete offers what they call "distributed testing," but after using it in a free trial I'm not real impressed with how it works. It just doesn't seem to have this use case in mind. I'm still working with SmartBear to see if there's something that can simplify what I'm trying to do.

My question: can anyone recommend a tool for testing a suite of applications across multiple machines, assuming that a test on one machine might require an action on another machine?

I realize I should test in isolation and simulate/mock the other machines, but the communications between the machines is complicated enough that it's a bit of a non-starter for me. I can revisit this if there aren't any great tools for doing this, but it no longer becomes a good integration test in my eyes. I do intend to simulate the vendors' data feeds since they are generally simple and there's no other option since we don't have their software.

share|improve this question
1  
When you say 'integration tests', do you mean dev-integration tests or black box acceptance tests that span across multiple applications? –  Suchit Parikh Aug 6 '13 at 1:15
    
The latter. I'm looking for the lowest barrier to start testing and I figured that would be to test the system as a user would use it, then work towards better test isolation from there. –  Patrick Quirk Aug 6 '13 at 12:12
add comment

2 Answers

I'm not aware of a tool that's capable of doing this across multiple machines. I agree with you that TestComplete's capabilities in this regard are less than ideal. The master-slave setup is fragile at best.

What I used to do in my previous job (with TestComplete) for a similar kind of situation was:

  • As many of the applications as possible went on one machine. The configuration was more complex as they were designed to operate on different systems, but it could be managed.
  • Wherever possible, communication that involved a common database was tested by checking the database rather than by multiple applications.
  • If multiple machines were necessary, we used the SysInternals tools to invoke the required applications on the other system and treated them as 'dumb' for the purposes of the test: data would be sent and the returned data captured but no attempt made to monitor the remote application. After the test completed the remote application's log would be inspected for errors.
  • Where possible, communication would be sent directly to the target application via the API rather than invoked from a different application.

We chose those methods because the more moving parts you have to hook together for a test, the more points of failure you've got. The few times we used TestComplete for this, we found we had more false positives than actual bugs caught, where the API-based and single-machine setups typically caught any regressions in the communications between applications.

I know this isn't the answer you're looking for, but it is a potential solution to your issue.

The protocol for communications between your applications isn't exactly undocumented - the code is the only documentation there is. You could build your own documentation via code inspection, port sniffing to capture the traffic between applications, any logging of messages sent/received and so forth, and use that information to generate an API-based test (which would be preferable in this situation - fewer possible points of failure).

share|improve this answer
    
Good suggestions, thanks for the detailed response. I could see something like that and getting it to work, though querying the remote machines will be difficult (they're pretty opaque). I assume the SysInternals tool you mention is psexec? –  Patrick Quirk Aug 6 '13 at 12:16
    
Yes, psexec is really useful for this kind of thing. You do have to have everything preconfigured beforehand, though. –  Kate Paulk Aug 6 '13 at 12:17
add comment

I prefer to stay free of certain tools and rely more on the programming language. I feel the BDD style does a pretty good job of driving the tests and documenting.

I work on a system that spans across multiple daemons, and we are currently working on building black box acceptance tests. We are using python as the core programming language with lettuce as the BDD framework - documentation for lettuce here. All the major programming languages have a library or a package built up to support BDD.

You will be able to document your features (behavior/requirements) with your features files; and all the test work can be handled by the programming language. The normal UI testing can be handled with the likes of selenium, watir, etc. You can mock out any external dependencies.

As you are building black box acceptance tests, your tests shouldn't worry about where the actual applications are running.

share|improve this answer
    
I can't use selenium/watir since those are for web apps (correct me if I'm wrong). In this question I'm specifically focusing on the tool because there isn't one that I know of to do what I need. I'm familiar with BDD, but something has drive the UI across machines here, and as I said above mocking out some of these dependencies is quite difficult so I'd need to run multiple machines. I'm asking here if a tool exists that helps me avoid creating these mocks. –  Patrick Quirk Aug 6 '13 at 18:09
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.