What are the tiers of testing that should be done on a large scale distributed system?

Question

Let's say you have a large distributed system in the cloud that's got dozens of developers working on it and actively adding code to. Your job is to make sure all of the quality gates are used and the system is continuously tested at and after checkin for potential malfunctions. Assume you're also working in Agile and do TDD.

I'm honestly only personally familiar with tests such as unit-testing (which I myself use for TDD) and with integration testing, which is also mandatory if your piece of code has any sort of dependencies. I unfortunately do not have a good understanding of all of the other layers of tests which the test team at my organization has historically taken ownership of. I heard of build verification tests, smoke test, validation tests, acceptance tests, smoke tests, system tests, scenario tests and so on, but I'm not familiar with the specifics, and would like to learn more about them!

More importantly, I would like to understand which of the abovementioned tests are crucial for a successful development flow in projects of different sizes, and how beneficial one test is over another. For example, in a small 5-developers project, how many of the test types should I worry about having? Are UT and integration enough? How critical would it be to have a more advanced / broad test?

Thanks!

Answer 1

Couple of ideas to evaluate application architecture and decide Test Approach

Testing different layers

My approach to get started is evaluate (Web Service, Database Testing, UI Testing, Security and Cloud specific testing on infrastructure/scalability/performance)

• If it is SOA-based implementation, you probably would have a service layer exposed which can be tested independantly
• Database testing can be done independently for verifying schemas, procedures, business logic in procedures based on implementation
• At UI Layer, verifying layout, cross browser testing, end-to-end testing can be performed
• If you have any queuing based implementation (Biztalk / MQ Series / Service Broker) these can be independently tested. Post messages in queue and see how they are picked up and processed
• For cloud-based implementation, storage aspects, security aspects, performance and scalability need to be tested.
• For cloud based PaaS, SaaS service degradation over period of time, extended loads needs to be validation. Service degradation can happen for peak seasons when multiple application demand resources for extended period of time. This bottleneck need to be tested

What to test on different layers

Somehow orthogonal to layers above:

• Build Installation Testing, Functional Testing (Service/DB/UI layer), End-to-End Testing, Security and Latency. I do not mean exhaustive performance testing but at least to see how much network latency for users accessing from different regions.
• For SaaS testing see check SoftTest Webinars Testing a SaaS (Software as a Service) Platform in an Agile World, and SaaS Testing Overview - Foundation

• Important takeaway from the above presentations:

  • application related security (SQL Injections, Cross Side Scripting),

  • infrastructure related:

  • disaster recovery mechanism,

  • latency (Network, Data Transfer, Load balancing, Remote Monitoring). see related read.
• The more you have idea on architecture and implementation you would be able to categorise and apply different test approach listed above. I hope you have access to see unit test cases executed by the developers. You can also see how UI, DB development is tested. This was you would be able to see what is amount of test effort done by dev team in validating the functionality

End-to-end testing

On top of all listed approach, You would need to execute end-to-end test cases for all priority one business scenarios

Hope it helps.