Choosing Testing tools against multiple programming languages

Question

I have a few questions in regards to setting up/deciding automation frameworks against microservice architecture OR codebases that span various programming languages.

Specifically in this "scenario" a company is setting up automation against a suite of products that are interlinked from various services. Some of this may be older legacy code (C/C++/Java) and Desktop based applications, and newer services running as actual web applications and using something different (C#/JS(Node) with front-end libraries.

This presents an issue when it comes to choosing testing frameworks/libraries:

1. Picking something that's compatible
2. Not having members learn multiple different libraries/frameworks (Pretend all QA's have never done automation before)

On one hand, having the test codebase with the service/repo that it's testing against makes sense, but it makes you want to use similar libraries.

Another option is choosing something that sits outside and hit's API's/UI without having any coupling with the code (But you lose out on some functionality, such as using models/classes/functions from within the code perhaps). But you get the benefit of being language agnostic.

I realize this may seem to be opinionated, but I imagine there is a "best practice" with why and why not. Also I am curious how others have handled this scenario.

Answer 1

I would approach it from these perspectives:

1. What kind of testing are you doing ?

Is it at the unit, integrated or end-to-end level? Is it white, black or grey box?. If it is at the unit or integrated level and you want to call existing routines in application code, this obviously lends itself to advantages from using the same language. If it is at the end-to-end level through a browser then it may make sense to use a different language. For example, currently Cypress.io is used a lot for browser testing so that means Javascipt, regardless of the backend language / frameworks.

2. What are you using for the backend language?

For example if you are using Python than you might consider python-selenium-webdriver to be a good choice for end-to-end, whereas if your backend is React (Javascript) then cypress.io (Javascript) may be a better choice.

3. What are the team dynamics ?

Are QA engineers embedded on application development teams and able to pair with fellow language developers, or are they segregated into a separate team where their pair is usually another QA person. The former makes a stronger case for sharing the same language for code.

4. What capability do you need from the testing tool?

For example for browser testing, cypress.io typically gives a much better experience than using selenium so the features and capability of the platform may outweigh the advantages of sharing a language.

5. How are you organized?

Are QA's paid as much as devs? Are QA's involved in all planning activities? Do QA's review application code for bugs? The more involved QA's are in application development, the more reward comes from sharing a language. Also, if QA's are second class citizens they will typically use the programming language learning opportunity to become a developer and get paid accordingly. Or they leave to get the same benefits elsewhere. This can led to a low quality automation codebase continually worked on by newbies.

6. Are different programming goals recognized ?

Application code and automation code share some quality characteristics but their priority can be quite different. Application code can have a focus of being performant and both memory and time efficient and may use extensive optimization to do so. Automation code on the other hand tends to focus more on readability (by treating the tests as the specifications) and how well application code is being exercised. It is important that these differences are recognized or the wrong foci will be used when programmers from one discipline pair with those from another

7. What structures are in place for pairing ?

Pairing can be a new or difficult practice and when performed by specialists from different areas such as dev and test, there are many organizational, culture and possibly geographic barriers to it succeeding organically. Also programmers may be introverted and not want to share by default. Leaders, coaches and guides are essential to make sure that collaboration happens in practice and not just 'because they use the same language'.

Answer 2

We developed WebTau to have an easy to use API to write e2e tests across UI, REST API, GraphQL, WebSocket, DB, CLI.

You can write tests in Groovy but tests look like a testing DSL, so you don't even need to know it is Groovy.

Or you can write tests in Java using JUnit.

We used Groovy tests to test micro-services written in Go, Python, Java and C++.

scenario("my bank balance") {
    Alice {
        http.get("/statement") {
            balance.shouldBe > 100
        }
    }

    Bob {
        http.get("/statement") {
            balance.shouldBe < 50
        }
    }
}

Answer 3

“You can do anything, but not everything.” - David Allen

Go for full blown Contract(API) testing as common denominator & low hanging fruit

Use API testing to target integration/contract testing between all the old & new subsystems to test the interlinking.Just focus on the different contracts between subsystems and you will go a long way with minimal effort.

WHY?

• Its common denominator - Its based on microservices architecture so all systems/languages understand
• Language Agnostic - easy to learn as single thing for QA
• Higher Level- Integration Testing -assuming all subsystems will have "something" on unit level in their own language.It will be good idea to test various contracts set up between them.
• Future Maintenance - As it will be language/UI agnostic , it would be easier to maintain in the long term even though implementation(UI/framework/language/backend/tooling) of subsystems drastically change but contracts between them remain same.

Answer 4

One language independent technique that I have used with great success is to produce a script configurable, event driven, test tool that would respond to an event such as a time elapsing or a message arrival with evaluate, log & optional message send. Since the system under test consisted of an input gateway an output gateway and an inter-process protocol it was only necessary to implement the 3 interfaces (plus a master controller) and any sub-system could either be emulated & substituted.

This allowed any of the sub-systems or any connected collection of subsystems up to the entire system to be tested in isolation by surrounding it with instances of the emulation. The master controller was responsible for starting, logging and shutting down cleanly. It also allowed debugging of the system by substituting one, or more, of the subsystems with the test harness and the, by then existing, behaviour definition(s).

Since a common format was selected for the response table definition and that format closely aligned with the way that system was specified it was quick and easy to define the expected responses to both valid and invalid events. Also any developers or testers that were working on components that were logically close to the ones that had already been addressed were able to re-use the existing response tables.