3

I'm in charge of leading a QA Team and one of our tasks is to develop Automated Microservice Tests.

We've been successfully testing these Microservices for a year. In summary, our Testing Process is:

  1. Review the documentation
  2. Usually the Microservice output will vary accordingly to ~6 input combinations. We write the Specifications for those ~6 Test Scenarios
  3. Produce messages with our Test Data to the source Topics using a REST API
  4. The Microservices will consume and process these input messages, map the fields, aggregate/transform the data, perform calculations and produce the output messages
  5. Consume the messages from the output Topics using the REST API and perform the assertions

Additional info: The Developers are responsible for the Unit Tests. The QA Team is responsible for the Functional, Integrated, Performance and Exploratory Tests. Our Automated Tests are already integrated into the CI/CD Workflow.

However, the company is migrating from Docker to Kubernetes, we will use a different REST API, so the Tests will be refactored. For this reason, before refactoring the DevOps Team threw an idea around, what if we changed the Testing Paradigm and, instead of developing Automated Microservice Tests, the QA Team would develop "Testing Microservices"? These "Testing Microservices" would run in the Production Environment and validate all generated messages. In my opinion this goes against all good Testing Practices but I'd like to hear your opinion.

  1. Developing Microservices is a Development Task, not a QA Task
  2. Would we have twice as many Microservices in Production (it would make more sense to run these "Testing Microservices" in Staging but they said Production)? This would greatly degrade the performance of the server
  3. What's the point? I don't see any benefit in doing this, only drawbacks

Plus, our Automated Microservice Tests could also do this, we don't need to develop "Testing Microservices". But again, there are only disadvantages. Each Microservice generates tens of thousands of messages daily. Why would we perform tens of thousands of Requests to test each Microservice, when the Test Scenarios have already been identified and the output will vary accordingly to half a dozen input combinations? We can do the same task performing only 6 Requests, thus avoiding the degradation of the server's performance.

6
  • 1
    What was the rationale for the "Testing Microservices" solution? What advantages over the current solution were presented? Nov 14, 2020 at 17:19
  • 2
    Why would someone downvote with out any valid feedback, i upvoted
    – PDHide
    Nov 14, 2020 at 20:19
  • @wiLLie22 i think what DevOps team meant was contract testing ? what you mean by " source Topics"
    – PDHide
    Nov 14, 2020 at 20:31
  • By source Topics I mean input Topics
    – wiLLie22
    Nov 14, 2020 at 21:19
  • 1
    There is no point in answering before you check what are those testing Microservices, in the meanwhile please read [Martin Fowler][martinfowler.com/articles/microservice-testing/#agenda] and [Cindy Sridharan][copyconstruct.medium.com/…
    – Rsf
    Nov 16, 2020 at 14:12

2 Answers 2

5

There needs to be little more clarity on how the team currently works and what they expect to be working as.

By reading the question it looks like the current Test strategy is like :

enter image description here

Huge drawback of this implementation:

  1. This takes away the basic pillar of microservice architecture, this test creates a dependency between microservices.
  2. You will not be able to test the microservice implementation. Here even though the architecture follows microservice rules, the test strategy still follows a monolithic approach of testing the product as a whole.
  3. Even if one module fails the test fails to test other modules

What DevOps team recommends is :

enter image description here

Advantages of this:

  1. You will be testing individual microservices
  2. You don't have to develop microservice but create mock servers that mimic response that the test target microservice expects other component
  3. This strategy ensures each module works as expected and works when integrated together
  4. Removes unwanted dependencies and adapts as microservice testing strategy

Recommendations:

You can use wiremock if you are using Java. But i recommends to migrate to postman as it supports mock servers inbuilt and it is really easy to use

https://blog.postman.com/postman-engineering-microservices-example/

8
  • 4
    Please comment when you downvote make your vote helpful for others who reads it
    – PDHide
    Nov 14, 2020 at 21:10
  • 1
    Thanks for taking the time to write such a detailed answer. I agree with you, our current approach is similar to the one you described. There's a Kafka Environment for testing, the same Microservices that are running in the Production Environment are running in the Testing Environment. We have a set of Automated Tests for each Microservice.
    – wiLLie22
    Nov 14, 2020 at 21:26
  • 1
    Basically there was a Microservice in Production that had been working properly for months. Suddenly it started performing wrong calculations. DevOps wanted us to validate all generated messages, so that we could detect when the Microservice started to malfunction. Turns out the Microservice started generating wrong values when DevOps deployed a new version (1.7) to Production. The QA Team had only tested version 1.6.
    – wiLLie22
    Nov 14, 2020 at 21:40
  • 1
    We are testing individual Microservices, our current approach is similar to the second one you described. In my opinion, it doesn't make sense to validate all generated messages (tens of thousands per day per Microservice). Our current code is already performing testing using all relevant input combinations (performing a few tens of Request at max) without blowing up the server.
    – wiLLie22
    Nov 14, 2020 at 21:46
  • 1
    Exhaustive testing is not possible so you just have to figure out critical use cases using boundary and pair testing techniques . If you already have second setup then there is nothing much to improve
    – PDHide
    Nov 14, 2020 at 21:53
4

You test micro services the same way you test all software, with Unit, Integrated and UI tests. As PDHide details nicely, a key part of micro-services is that they give you the ability to perform unit and integrated testing without all the dependencies that traditional testing would require. It is up to quality software engineers to both recognize and implement this. It won't just 'happen' and traditional business will often not recognize this technical distinction and need.

The question you seem to be asking is perhaps more

Should we test in production?

That is a great and very relevant question.
Companies that release hundreds of times a day obviously can't spend a traditional amount of time running and verifying slow and brittle end-to-end UI tests for every use case scenario.

Well actually many do, but it is a nightmare, the value is greatly diminished by the blocking effect and slowness to get feedback to the developers. This may be accompanied by unhealthy work environments that are characterized by finger pointing and blame assigning. The poster on the wall may say we're all one big team but the daily interactions may not reflect that fully.

Initially, automation over manual testing seems to address this (as vendors loudly proclaim) but as automation suites with thousands of tests are implemented, the bottleneck returns, though now for different reasons that are now about the automation itself. After a while test suites take a day, or more, to run. Not good. The startling thing about this is that companies who have figured this out are not small, one-off startups, but rather the industry giants of our time - Yahoo, Facebook, Netflix, Amazon, etc.

In order to do this that are two main practices:

  • Do most testing before you reach the UI layer, using the Agile Testing Pyramid as your guide.
  • Monitor and measure production to see what real users do, using AB, Canary and Feature Flag testing

On re-reading your question, one idea comes to mind - maybe testing micro services would work if viewed specifically as the integration test layer in the Agile Testing Pyramid. Neither replacing or duplicating the extensive unit tests that accompany the code as it is being written (TDD) or the selective UI tests (ideally through a BDD process). This does raise an interesting dilemma though - if you are rigorous about writing tests for functionality your testing time may grow endlessly unless you constantly refactor at the test level itself, e.g. move UI tests to Unit tests, mock out more dependencies, remove tests that never fail or are covered by others. etc. This can lead to approaches such as a fixed test run time but this is a more advanced tropic.

1
  • 1
    +1 thanks Michael for suggesting overall shift left testing. Nov 16, 2020 at 16:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.