Test automation for microservices architecture

Question

I am in charge of implementing QA processes and test automation for a project using microservices architecture.

Project has one public api that makes some data available. So I will automate API tests. Tests will live in one repository. This part is clear to me, I did this before in other monolith projects. I had one repo for API tests. And possibly another repo for selenium tests.

But then here the whole poduct consists of many microservices that communicate via restful apis and/or rabbit queues. How would I go about automating tests for each of these individual servicess? Would tests for each individual service be in a separate repo? Note: services are written in Java or PHP. I will automate tests with Python. It seems to me that I will end up with a lot of repos for tests/stubs/mocks.

What suggestions or good resources can community offer? :)

Additional info: Product is being developed by one team of 6-7 devs and 1 qa (me). We are planning to implement CI and CD also. On pull request components should be deployed to a test env and my tests executed there. Then if successful components will be deployed to staging env and some smaller set of [smoke] tests might be executed. In a case of a successful deployment to stag env and successful smoke test execution components might be deployed to production. All this done by CI/CD tools and tests that Im about to write.

Answer 1

• Hopefully the Developer teams know their responsibilities i.e. having high test coverage at Unit and Integration level (Test pyramid). Assuming that, you can focus on high level testing i.e. end2end testing and contract testing.
• Contract Testing: A very good practice in micro-service architect.There are some solution for it. One of the most suitable one is a consumer-driven contract testing called "pact".
• If possible, keep your Test Automation in the same repo since you may need Mocking/Stubing plus versioning can be tricky. Also staying in the same repo could be a help for you CI/CD pipeline.
• Have a look at Microservice testing by M.Fowler. It gives good idea about different layers of Micro service testing.

Answer 2

What to test and how

• You definitely need to test full flows of the system, to see whether services understand each other, in particular, when one of service responds with error messages. This is a place where I usually find problems. Mocks will definitely help here.
• Also, if you're working with queues, then many there might asynchronous events and time-outs that may come into play. You can try simulating each such scenario individually but performance tests when many concurrent threads are loading one of queues with unexpected number of messages can reveal interesting bugs.
• Many scenarios are easier to automate at unit testing level, where you do not need to start an HTTP server to host a service or service mock. This is particularly the case for negative scenarios. I usually work closely with developers of such services and every time I find automating a scenario is to expensive at system level, I ask developers to add a test to their unit testing suite.
• Martin Fowler identifies multiply levels of testing for microservices: unit tests, integration tests, contract tests, and end-to-end tests. However, make sure you're really having microservice architecture there. I've seen many microservices ending as distributed monolith, sharing code or even sharing databases. This problem has been described in InfoQ.

How to organize your repository

It is not only matter of your test strategy but also how you work as a team.

• Are microservices owned by different teams? Is the code for those microservices in the same repo?
• Who is going to maintain and update those tests? Only you, other testers or also developers of microservices?

I usually prefer to have tests close to the code they test, so if anyone new comes to the project, she can start automation with one or two code checkouts.

• How will you launch those tests in a development pipeline? Do you want to have one single status whether all tests passing or failing, or you prefer to launch first tests for single services and only if they pass launch end-to-end tests?

You do not need separate repos for each group of tests. Packages enable organizing structure of tests in a way you wish, so it is easy to grasp test coverage looking at your packages names and structure. Then there are test groups that can be used to annotate your tests. They will ease launching separate groups of tests, especially if you want to have separate jobs in your CI server (e.g., Jenkins).

• Whether to have a separate repository for service stubs, mocks and other helping tools is another question. Are you going to use those tools across other testing projects as well? Are other people going to use those tools outside of your project?

I usually prefer to start small, with one repo, well structured. Creating many repositories from the beginning makes maintenance unnesessary complex. When the repository starts growing, you will need to refactor it from time to time, update the structure. If refactoring is not enough then you can think whether to move some tools or tests to separate repos.

Answer 3

You mentioned "rabbit queues", i.e. RabbitMQ. There are a number of ways to configure RabbitMQ involving trade-offs between performance, fault tolerance, and resource requirements. I suggest that you read the RabbitMQ Reliability Guide, and then read Aphyr's post on RabbitMQ, and then think about how that might apply to your system.

Here are some things to test:

• What happens if a publisher is unable to publish a message because the RabbitMQ server is too busy or otherwise temporarily unavailable?
• What happens if the publisher writes messages much faster than the consumer can read them?
• What happens if a network partition causes RabbitMQ to go into split-brain mode? When the partition heals, there may be two master nodes, and one of them will decide to truncate its data, resulting in lost messages. You can't do anything about RabbitMQ's behavior, but you can try to determine how catastrophic that will be to your system. If you fail to send some email's, that's one thing; if you fail to deposit money in someone's bank account, that's something else.

Answer 4

Using Micro Services is a growing trend.

I would look at the big picture for guidance on what to test where

You can use the use the 4 quadrants approach, they are:

• Unit Testing. This will cover basic tests for the api and within each service. Code that uses other services should mock and stub them out.
• Integrated Testing. This is where you specifically test services from one app to another. You'll need to to create test versions of the services that given requests will use.
• Performance. You should look to create a test system that reflects the production system. This means creating test versions of all the services under test and volume that reflects real usage.
• Exploratory testing. This is where you can see if all the services tie together correctly from an actual users perspective.

Answer 5

In all honesty, I would write the automation in the same language as the product under test. I find this reduces the barrier between dev and qa (in environments where the roles are separate). Furthermore, you can easily pull in the code under test.

That being said, I'd recommend keeping the code in the same repo if at all possible to ease versioning pain. Ideally, you would make these tests part of the actual project.

If you did go with Java, I can recommend the libraries Rest-Assured and Restito. I love these libraries when testing against APIs.