19

If an application is written in a certain language, e.g. Python, Ruby, Clojure, etc. should Quality Engineers use the same language to write the tests?

Should this be a decision that we can apply to multiple applications written in different languages?

What factors should be considered and what are the advantages / disadvantages of using the same vs different language? How does this relate to white box / black box testing ?

  • 1
    It is advantageous to avoid technology proliferation and ensure QA engineers and Application engineers use the same skills. – Martin Spamer Sep 2 '17 at 17:12
11

As Michael said, there are a lot of advantages to the test team using the same language for their automated tests as the application language. About the only point Michael missed is that when both application and test automation use the same language, developers can help testers with automation code, allowing things like tester-developer pair programming while building tests (at all levels).

Multiple applications in multiple languages - I'd consider going with one language for the test automation so I can easily re-use any helper routines I've created.

Black box vs White box testing - The choice of language for test automation isn't relevant to white/black box testing: what matters there is whether or not your test automation is examining the internal logic of the software you're testing. If you're testing external logic, such as selecting option A disables option B, it's black box. If you're testing internal logic such as calling function X with input value Y should call function Z, it's white box.

8

My preference is as following:


Primary: Use the same language when practical

Advantages:
- code reviews can be shared for both app and test code
- automated UI test writers can easily understand what unit tests cover
- discussions use the same terminology without constant translation
- reuse of infrastructure parts without change

Disadvantages:
- testers may not have skills in the language used by application engineers
- testers may lose their independence
- testers may think more in terms of application code and less in terms of user experience


Secondary: Use existing testing language expertise

Advantage:
- automation developers can apply existing expertise and knowledge

Disadvantages:
- code reviews are harder
- easier for devs and testers to have a gap between them
- application and test code bases need to be kept in sync


Tertiary: Use a default selenium language binding across the organization

Advantage:
- automation developers can apply existing expertise and knowledge
- reuse of techniques and practices
- automation engineer can reuse code
- easier to develop and share wait techniques

Disadvantage:
- code reviews are harder when the application code is in another language

  • From Primary I agree only with first item from advantages, from Secondary the same, only advantage part and the same for Tertiary, all other items does not seem so strong to me to consider. – lauda Aug 17 '17 at 19:47
  • I don't see Selenium in the original question – Rsf Aug 18 '17 at 18:42
6

First level of your defense are unit tests, and they are written in your app language by definition.

For QA and automated integration/GUI/API/e2e tests you want to use the most productive language you can find, because (let's face it) test are not the billable code (even if without tests you cannot produce reliable billable code). I find I am more productive when using dynamically typed languages, like Python (and I have decades of experience using more than a dozen of different languages, so I earned my right to have opinion here).

So using the same language as app is beneficial if that language is optimized for productivity, and QA/e2e/AutoTest developers are really productive in it.

OTOH is app language is optimized for performance (C, C++ and other statically typed languages), it might make sense to use language like Python for automated e2e tests, to keep the productivity up. Or for Angular app, you will use JavaScript in front-end but might use another language in backend. So makes sense to write e2e AT in JS, not in backend language.

  • "First level of your defense are unit tests, and they are written in your app language by definition." – I know several Java projects that use Ruby, Scala, or Clojure for Unit Tests. (And many projects use a Java-based internal DSL.) I also know of at least one embedded project written in assembly + C that uses a self-developed testing and mocking framework written in Ruby to write unit tests in Ruby. – Jörg W Mittag Aug 17 '17 at 19:24
  • Interesting. I says a lot that language is not good even to write unit tests for own code. Java vs. Scala or Clojure are different, IIRC they all use JVM so can call each other. For embedded project, I would use Forth which flexible enough to write both core code and unit tests (and more compact than assembly, in my experience) – Peter M. Aug 17 '17 at 20:44
3

When to use the same language as the app uses:

  • you have enough technical qa's that know this language
  • you have/chosen a stable automation framework for this language
  • dev's can help in writing automated tests
  • dev's can help in writing additional libraries for the framework if needed

Why reasons to not use for choosing the same language as the app use:

  • programing language is not so known, too old or too new
  • you have skills in the dev team, this will not help qa
  • dev code review will not help much if they don't know the framework, what those methods do/return and what validation should use for that
  • dev can understand code reviews, will not help if test are poor written and not efficient/maintainable
  • reuse of code from dev's, this will happen rarely or not at all, in the bast case dev's can help with a push forward when needed, but not in the log run
  • you don't have enough technical qa's that are proficient in this language, if they are to few you might end with some code that you can never reuse and/or maintain

Our days.
Apps are written in multiple language and those languages should not influence you in choosing the right tool and the right language.

You need to select few tools that you can use for automation, and choose the one that use a language your QA team has and also offer the rest of the things you need (reporting, etc.)

If you want everything to work smooth then:

  • create a coding standard and also use it
  • educate your team to know how to use the tool
  • always improve, make sure you reuse as much as possible
  • use a design pattern and you have a strong architecture in your framework
  • keep a clean code, do weekly automation meetings
  • put someone in charge, like a framework/automation owner
  • code reviews are very important, make sure enough people are doing the review and from them the key people like automation owner or a senior automation engineer

Plan > Prepare > Perform > Perfect

2

There are many things to consider about automation test. What kind of tests are we talking about? Do you want to write end to end tests, unit tests? Are you working on a mobile or web application (maybe both)?

Moreover do you work with other QA on automation or are you alone, do you have a good experience in programming or not?

If you're not comfortable with a specific language maybe you can do your tests in the same language as them (if possible) so if you need help or some programming tips, it can be a plus.

But if your company doesn't specify any language for your automation tests, just use the language you feel comfortable with. There is no need to get headache on something you're the only one to manage.

2

There is no mandatory use of the application specific language to write tests. I've worked at places where application is .NET - my automation framework in Java; application is Java - my automation framework is .JS based. From my experience, picking up the programming language to do E2E tests really depends on the several questions:

  1. What programming language am I most comfortable with?
  2. Am I enforced to use some specific programming language at my job due to the management / leadership choice? (open source VS paid tools)
  3. How much help from the dev team would I need at the test scripting process if any impediments occur?
  4. Am I in position to suggest what programming language is best for E2E test? (good online support, big community of users and so on)
2

It depends on who the (human) audience is for the tests. Typically I use two.

At the big picture end, the best way for you and your client to be confident you have understood requirements is to have tests that can be read as worked examples of each user requirement in as close to English prose as possible.

An example of this is back in the 90's I took the "testing notes" (which took my sales boss around 80 hours to do all the checks) and wrote a test framework that could understand a simplified (and rationalised) set of instructions that read like English, so where he wrote "click Such and Such button" and "Enter 'some text' into Author field", that is what the test script said.

Now days you would use something like https://cucumber.io/ which you can use to test systems written in other languages.

Tests that only the technical people care about the specific details should be written in what is the simplest to write and clearest to read by the technical team (Eg If the client asks for a sorted list, they won't generally care about how it gets sorted, just that it is sorted). The closer to unit testing you are, the more it makes sense for the test to be written in the same language.

The xUnit test framework is one of the popular frameworks for unit testing in a multitude of languages.

2

A side point that I think has been missed is the importance of ensuring that interface & communication tests are written in a platform agnostic language and manner.

I recently had a case where the writers of a utility that was to communicate with our main application complained when it failed the communications tests. The tests were written in Python using the struct library which allows you to specify the structure including specific lengths, byte ordering and packing.

They thought that there couldn't be anything wrong with there protocols because they were using the same .h files as the main application but had not accounted for the fact that they were developing with Visual Studio for a MS-Windows target and they needed to be able to talk to the main application which was built with qcc for qnx on a big endian platform.

If the test suite had been written in the same language this problem would have been spotted a lot later.

So for anything that communicates with other platforms you need an independent, often different, language.

0

If possible yes. The main reason for this is because context-switching is expensive. If you have to work in multiple languages (and maybe even in multiple IDE's) then for one feature your team might be switching back and forth between tools and languages. Each switch will cost time, certainly if one of the languages is only used once in a while.

Being proficient in a programming language is hard, being proficient in multiple is even harder. The less you need to switch thinking the better. Also you might not fully use the features of the second language and create shortcuts just to get it working and be done with it.

There are good reasons to introduce extra tools or languages to a project, just make sure you weigh the cost of context switching, learning curve, motivation to learn (mainly from developers) and added complexity for new-comers to your project.

Working with developers:

As test engineers are often the lesser coding gods, being able to utilise your lead developers to assist with good coding practises on the test automation is must in my book. Therefor make sure you pick your test automation languages with them together.

I would be really careful with introducing an extra language as testers into a project if you expect the developers to run, maintain and extend tests as they are adding new features. Just make sure you introduce testing tools with a whole team approach.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.