We have got an old system "Qwerty1" - that makes a lots of things,
then we also have got another system "Querty2" it's actually a remake of Qwerty1
(for several reasons) Notice we still are live with "Qwerty1".

now we want to to be sure for all tested indata in Qwerty2 we get the same outputdata in the both systems.

What is this methodology called?

Thank you in advance

  • What do you think should be an appropriate name for it? – IAmMilinPatel Jan 9 '18 at 10:12
  • to be honest I have no idea really I have been asking my colleagues but we are not sure. – XsiSec Jan 9 '18 at 10:12
  • The testing part is covered, but the software development method used is called "Big redesign in the Sky", for several reasons the new product never gets delivered, as it is never feature complete as the old system still has to move forward as well. UncleBob nicely explains these situation in his free video: cleancoders.com/episode/clean-code-episode-1/show – Niels van Reijmersdal Feb 6 '18 at 13:13

We would tend to consider this a form of regression testing as you are testing that the software still behaves the same as it did before.

Regression testing is a type of software testing which verifies that software which was previously developed and tested still performs the same way after it was changed or interfaced with other software.

Qwerty1 and Qwerty2 may be separate products, but consider them just to be two different builds of the same functionality and the testing considerations are clearer.

  • That was my thought from beginning though I have never literally understood regression testing in that way even if there is a new product that perform the same functionality as the old one. Or? – XsiSec Jan 9 '18 at 16:49
  • 1
    Any time you're checking that the behaviour of something hasn't changed - that's a regression test. Even if you rewrite the entire codebase and rename the product - if you compare it to an old version (such that a difference in behaviour is a failure), that's regression. Note that in perfect/ideal regression testing - you expect to find the same bugs as before, there should be no change to behaviour at all (but this is obviously not realistic). – Bilkokuya Jan 9 '18 at 16:53
  • 2
    I wouldn't agree with you. Regression testing is conducted on the same application to identify not desired 'mutations' and side effects. In case of OP, they are testing 2 different applications which are built for the same purpose (i.e. have the same functionality). – olyv Jan 9 '18 at 17:18
  • 2
    @olyv I'd argue it's conducted on any new version of software that adheres to the same requirements. It doesn't matter if the change was a minor patch, or an entire new product. If you wish to test a new piece of software runs identically to an old one, you perform regression testing - using the same test cases on each and considering a pass to be whatever the result of the old software was. – Bilkokuya Jan 9 '18 at 17:31
  • 1
    Why overthink it? You want to verify that some new code has the same behavior as some old code = regression testing. – Alex Jan 10 '18 at 0:23

It sounds like you're talking about Comparative Testing (also known as Product Testing)

...the process of measuring the properties or performance of products.

  • I added a critera to the question, I am not sure if that was clarified enough before. – XsiSec Jan 9 '18 at 11:47

I would go with Benchmark test. http://glossary.istqb.org/search/benchmark%20test

(1) A standard against which measurements or comparisons can be made.

(2) A test that is used to compare components or systems to each other or to a standard as in (1).

  • I added a critera to the question, I am not sure if that was clarified enough before. – XsiSec Jan 9 '18 at 11:47
  • @DanielElmnas I still assume 'benchamrk' is appropriate term to apply. Because this is only about comparison of two systems – olyv Jan 9 '18 at 12:02
  • The original question has been edited such that my answer is now incorrect. The original Q talked about performance and the reason Qwerty2 had not been released was because it was slower that Qwerty1. The original question was more about performance rather than functionality. – Chris Adams Jan 10 '18 at 10:53
  • still, 'benchmark' doesn't strictly specify 'only quantitive measurements'. It's only about comparing – olyv Jan 10 '18 at 14:54

I agree with Olyv (and have 'up voted' his answer).

Qwerty1 is your benchmark. Qwerty is failing because it's not as fast as Qwerty2.

You ought to log the results for each test side-by-side so you can time how performance changes over time.

  • I added a critera to the question, I am not sure if that was clarified enough before. – XsiSec Jan 9 '18 at 11:47
  • Yep, it was clear. I was just reiterating based on your Q and Olyv's answer. I'd still call it benchmarking as Qwerty2 is a remake of Qwerty1. So, Qwerty1 is the benchmark perf that Qwerty2 is striving to better or at least match. – Chris Adams Jan 9 '18 at 12:01

It seems to me that you are refactoring

Looking to improve the performance of the code and the implementation of the design without changing the functionality.

  • Yes it sounds more like that, but is that a still a software test methodology? I did read following thread softwareengineering.stackexchange.com/questions/233222/… though this feels more like you maintenance the existing system in our we made a totally brand one that have same functionality more easygoing and better performance. – XsiSec Jan 9 '18 at 11:55

This could be considered a form of Canary testing, which is typically reserved for a DevOps context of slowly switching traffic from the old system to the new system, and monitoring IO for discrepancies.

  • It feels like few things we does it's Canary testing but less than the half it feels like. – XsiSec Jan 10 '18 at 8:39

The old system "Qwerty 1" will be the reference for testing new system "Qwerty 2", so the appropriate methodology "Comparison Testing",but it is not enough ,you will use this with other methodologies and techniques in SW testing to ensure the new one works fine with no(incorrect or missing function,interface errors,performance errors,business errors)


While not a traditional UI/website A/B test, I don't think it'd be wrong or misleading to say you're doing a blackbox A/B test and comparing the output, performance, whatever, between the two implementations.

I like the term benchmark, but not benchmark test. Clearly, Qwerty1 is the benchmark and the results of running it are the benchmark test using the ISQTB definition, but the actual compare/contrast of Qwerty1 to Qwerty2 is not a benchmark test.

I also dislike the ISQTB definition of benchmark test, mainly since in my experience, when people discuss benchmarking, it's often performance testing related, when it seems that your testing isn't focused on performance testing.

  • I don't agree that is somehow any kind of Blackbox testing its wrong in my opinion really. softwaretestingfundamentals.com/black-box-testing – XsiSec Jan 10 '18 at 8:37
  • Sure, this could be white, grey, whatever. I mentioned blackbox since that's all that's required if the goal is "to be sure for all tested indata in Qwerty2 we get the same outputdata in the both systems." In other words, all you need to do is try various inputs and check the outputs, the underlying details aren't important. – ernie Jan 10 '18 at 15:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.