Test strategy for a backend engine moving from legacy to latest Java stack - interview question

Question

One of my interview questions was:

Assume you are working on a desktop application. Client application communicates with Backend engine for biz logic and as well rendering of data. Come up with test strategy considering that backend engine is moving away from legacy technology stack to a latest Java stack.

It was evident that my answer was not satisfactory. Can you please suggest what would be a better test strategy for this case?

My suggestions were: Since the entire backend engine was migrated, we need to execute all of the system test cases and the automation scripts.

They were expecting the list of risks that may arise and the mitigation plans.

Answer 1

You effectively answered the question with a test plan - a high level list of what testing you would do - when they were asking for a test strategy, that is, a high-level description of how you would approach a migration or rewrite of that scale.

The other answers have covered a number of good suggestions. I would add things like:

• What is the usage profile of the legacy application. Chances are it has a relatively small set of functions that get most of the use. These would be high priority targets to check for functionality, performance, and security.
• Can the new application be run in parallel with the legacy engine? If it can, a period of parallel operation should be part of the strategy, along with some form of automated or semi-automated checking of outputs.
• What is the performance profile of the legacy application. You will want to ensure that the performance of the new application matches or improves on the performance of the legacy app.
• Whether there is API test automation of the legacy application and whether it needs to be changed for the new application. If the legacy app has an extensive set of automated API tests, and the new app is not changing the API call signatures or expected responses, these can be used as part of the testing strategy.
• What are the essential must-not-fail functions of the legacy application? These are the functions that must be covered first and have automated regression built to cover.

With this information, you can prioritize aspects of the testing approach to cover the critical features first, most used features next, and other features as needed. You can also lay out strategies such as parallel execution, running automated tests against both systems, running performance tests against both systems, and running security tests against both systems.

That's roughly how I would approach an interview question of this nature.

Answer 2

Well, first, somewhere, there was a miscommunication about the level of test they were interested in discussing. You were focusing on the lower level "what would I do to test ..." and they were interested in knowing what risks there were. If that sort of miscommunication happened a lot, then there's a larger issue there than how you answered, possibly having to do with the role you were interviewing for.

Second, I'm not sure you understood the size of the question they were asking. What they're proposing to do is often described as "changing the wing of a 747 while the plane is in flight." They want to completely replace part of a system without changing other parts of it. What you suggested is regression/API testing, which is part of, but nowhere near, all of what is needed in this case. Many such projects fail because they're so hard to do for any system of a realistic size.

As risks and mitigation plans (which I wouldn't consider part of a QA/test role's responsibilities to come up with; risks, possibly, but not the mitigation plans; that is more of a architect/project planner responsibility), some things that come to mind:

1) Performance. Can the new application handle the same load as the prior application? How well does it scale, compared to the other application? Does it have different stress points? (i.e., old version started slowing down because it saturated the database I/O, new version slows down because it starts having to page to disk, or something similar). Depending on the results of the test, you might need to move to different hardware, shard the backend database, or redesign the new application.

2) Known errors/undocumented functionality. It's actually common for systems to have either bugs or undefined behavior, and for client applications to rely on that behavior. So your replacement system has to not just behave the same way as the spec says, it has to behave the same way the other system actually behaves. In order to test this, I'd probably want to implement some sort of record/replay functionality at the information flow level, and verify that for a given client input, the new backend gives the same response to live data.

3) Complexity. As I said; these efforts are very hard to do, and tend to fail. A better way to approach might be to rearchitect the current backend so as to make it less monolithic, and have more defined silos of responsibility. Then you can, instead of replacing the entire thing at once, replace it in pieces.

Answer 3

This kind of open-ended questions (and few follows-up questions) are excellent to separate those who have the experience with doing the stuff from those who try to fake it. So don't try to fake it (it is easily detected), and faking it is just wasting everyone's time (including yours).

Instead, describe similar situation where you tested changing part of the system, where you do have the experience. If you don't have any, tell about what you did.

During interview, they do not care how to solve any particular problem they asked about (if they don't know, they will hire external expert). They want to know if you have experience solving problems, and what is your approach - so tell them what you know how to do, and don't try to fake some experience which you do not have.

You will get much more suggestions about handling difficult interviewing questions at workplace SE.