Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It only takes a minute to sign up.
Sign up to join this community
If an application is written in a certain language, e.g. Python, Ruby, Clojure, etc. should Quality Engineers use the same language to write the tests?
Should this be a decision that we can apply to multiple applications written in different languages?
What factors should be considered and what are the advantages / disadvantages of using the same vs different language? How does this relate to white box / black box testing ?
As Michael said, there are a lot of advantages to the test team using the same language for their automated tests as the application language. About the only point Michael missed is that when both application and test automation use the same language, developers can help testers with automation code, allowing things like tester-developer pair programming while building tests (at all levels).
Multiple applications in multiple languages - I'd consider going with one language for the test automation so I can easily re-use any helper routines I've created.
Black box vs White box testing - The choice of language for test automation isn't relevant to white/black box testing: what matters there is whether or not your test automation is examining the internal logic of the software you're testing. If you're testing external logic, such as selecting option A disables option B, it's black box. If you're testing internal logic such as calling function X with input value Y should call function Z, it's white box.
My preference is as following:
Advantages:
- code reviews can be shared for both app and test code
- automated UI test writers can easily understand what unit tests cover
- discussions use the same terminology without constant translation
- reuse of infrastructure parts without change
Disadvantages:
- testers may not have skills in the language used by application engineers
- testers may lose their independence
- testers may think more in terms of application code and less in terms of user experience
Advantage:
- automation developers can apply existing expertise and knowledge
Disadvantages:
- code reviews are harder
- easier for devs and testers to have a gap between them
- application and test code bases need to be kept in sync
Advantage:
- automation developers can apply existing expertise and knowledge
- reuse of techniques and practices
- automation engineer can reuse code
- easier to develop and share wait techniques
Disadvantage:
- code reviews are harder when the application code is in another language
Primary I agree only with first item from advantages, from Secondary the same, only advantage part and the same for Tertiary, all other items does not seem so strong to me to consider.
First level of your defense are unit tests, and they are written in your app language by definition.
For QA and automated integration/GUI/API/e2e tests you want to use the most productive language you can find, because (let's face it) test are not the billable code (even if without tests you cannot produce reliable billable code). I find I am more productive when using dynamically typed languages, like Python (and I have decades of experience using more than a dozen of different languages, so I earned my right to have opinion here).
So using the same language as app is beneficial if that language is optimized for productivity, and QA/e2e/AutoTest developers are really productive in it.
OTOH is app language is optimized for performance (C, C++ and other statically typed languages), it might make sense to use language like Python for automated e2e tests, to keep the productivity up. Or for Angular app, you will use JavaScript in front-end but might use another language in backend. So makes sense to write e2e AT in JS, not in backend language.
PS - AT = Automated Testing.
When to use the same language as the app uses:
Why reasons to not use for choosing the same language as the app use:
Our days.
Apps are written in multiple language and those languages should not influence you in choosing the right tool and the right language.
You need to select few tools that you can use for automation, and choose the one that use a language your QA team has and also offer the rest of the things you need (reporting, etc.)
If you want everything to work smooth then:
Plan > Prepare > Perform > Perfect
There is no mandatory use of the application specific language to write tests. I've worked at places where application is .NET - my automation framework in Java; application is Java - my automation framework is .JS based. From my experience, picking up the programming language to do E2E tests really depends on the several questions:
There are many things to consider about automation test. What kind of tests are we talking about? Do you want to write end to end tests, unit tests? Are you working on a mobile or web application (maybe both)?
Moreover do you work with other QA on automation or are you alone, do you have a good experience in programming or not?
If you're not comfortable with a specific language maybe you can do your tests in the same language as them (if possible) so if you need help or some programming tips, it can be a plus.
But if your company doesn't specify any language for your automation tests, just use the language you feel comfortable with. There is no need to get headache on something you're the only one to manage.
It depends on who the (human) audience is for the tests. Typically I use two.
At the big picture end, the best way for you and your client to be confident you have understood requirements is to have tests that can be read as worked examples of each user requirement in as close to English prose as possible.
An example of this is back in the 90's I took the "testing notes" (which took my sales boss around 80 hours to do all the checks) and wrote a test framework that could understand a simplified (and rationalised) set of instructions that read like English, so where he wrote "click Such and Such button" and "Enter 'some text' into Author field", that is what the test script said.
Now days you would use something like https://cucumber.io/ which you can use to test systems written in other languages.
Tests that only the technical people care about the specific details should be written in what is the simplest to write and clearest to read by the technical team (Eg If the client asks for a sorted list, they won't generally care about how it gets sorted, just that it is sorted). The closer to unit testing you are, the more it makes sense for the test to be written in the same language.
The xUnit test framework is one of the popular frameworks for unit testing in a multitude of languages.
A side point that I think has been missed is the importance of ensuring that interface & communication tests are written in a platform agnostic language and manner.
I recently had a case where the writers of a utility that was to communicate with our main application complained when it failed the communications tests. The tests were written in Python using the struct library which allows you to specify the structure including specific lengths, byte ordering and packing.
They thought that there couldn't be anything wrong with there protocols because they were using the same .h files as the main application but had not accounted for the fact that they were developing with Visual Studio for a MS-Windows target and they needed to be able to talk to the main application which was built with qcc for qnx on a big endian platform.
If the test suite had been written in the same language this problem would have been spotted a lot later.
So for anything that communicates with other platforms you need an independent, often different, language.
If possible yes. The main reason for this is because context-switching is expensive. If you have to work in multiple languages (and maybe even in multiple IDE's) then for one feature your team might be switching back and forth between tools and languages. Each switch will cost time, certainly if one of the languages is only used once in a while.
Being proficient in a programming language is hard, being proficient in multiple is even harder. The less you need to switch thinking the better. Also you might not fully use the features of the second language and create shortcuts just to get it working and be done with it.
There are good reasons to introduce extra tools or languages to a project, just make sure you weigh the cost of context switching, learning curve, motivation to learn (mainly from developers) and added complexity for new-comers to your project.
Working with developers:
As test engineers are often the lesser coding gods, being able to utilise your lead developers to assist with good coding practises on the test automation is must in my book. Therefor make sure you pick your test automation languages with them together.
I would be really careful with introducing an extra language as testers into a project if you expect the developers to run, maintain and extend tests as they are adding new features. Just make sure you introduce testing tools with a whole team approach.
