Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It only takes a minute to sign up.
Sign up to join this community
The company I work for is moving towards CI and DevOps.
Currently we store our test data (mainly XML test requests) either on a network drive.
We want to move away from this. Looking for a system we can store our test data and test cases online, and can have multiple versions of test sets for each release.
We have GitLab, but does it make sense to store test cases and test data on GitLab? Is it really designed for that?
Are there any test data management systems you would recommend?
This may sound odd and may not be the answer you accept but I want to pass on my advice from the last 5 years of doing Selenium automation in several different companies from small local shops to national players. One of the biggest lessons I have learned from all these encounters is, surprisingly enough:
Avoid test data management at all costs.
Let me explain: Obviously you need some test data. Something has to be typed in all those form fields after all! What I have found is the pattern goes something like this: You start with a simple text name like 'TestLast'. Then you start extracting that into a separate data store (always with the 'but I am using it in multiple places' comment). After a while the test data starts to take on a life of its own. Worst of all the company starts asking for different scenarios and you start implementing different data scenarios for them. This starts to become complex very quickly.
So, my advice is avoid this journey. When you need a last name, hard code 'LastName' as the text. Doing it in multiple places is ok. Checking the value with a second hard-coded static value is ok. gasp! After all even if you use a variable of some sort you'll use that variable in every place. Yes having the variable in one place seems DRY but actually I have found provides little real benefit in comparison to the overhead that starts to get introduced with the dreaded "Test Data Management" issue. I've been there, done that and not repeating my previous path of complex test data is my advice now.
The reality of needing specific data to generate and test specific results is most frequently actually a code smell - you are using the front end to test logic that is usually implemented on the back end. Instead of UI tests you should have unit and integrated tests for that.
I agree with @MichaelDurrant with one exception: instead of hard-coding all constants as literals where used, define bunch of constants in each test.
Reasoning: if something changes, most likely you will need to change both code and test values, so best is to have them in the same file.
Version control for the code will conveniently carry also versions of your test data, relevant to the version of the code.
I would recommend to design and baseline QA database copy which can be refreshed/restored before every regression run.
This helps to keep data in known base state before any run.
This strategy worked well for me in multiple large automation projects.
Up to large extent I would agree with @MichaelDurrant and verify maximum no.of data exhaustive edge cases on lower levels of test pyramid.
you might thought of using data builder pattern for test data generation. or you can also design an algorithm to extract data(masked) from production server to fit the data providers for your tests.
