3

The company I work for is moving towards CI and DevOps.

Currently we store our test data (mainly XML test requests) either on a network drive.

We want to move away from this. Looking for a system we can store our test data and test cases online, and can have multiple versions of test sets for each release.

We have GitLab, but does it make sense to store test cases and test data on GitLab? Is it really designed for that?

Are there any test data management systems you would recommend?

  • David , is your question specific to any test types you are performing like unit,db, web services/API or UI Tests? – Vishal Aggarwal Mar 12 '18 at 11:03
  • 2
    Closing mafia: why you swarmed to close this perfectly valid, relevant and on-topic question? If you are too ashamed you write why you voted to close it, you should be too ashamed to vote to close it. Or you voted to close it just because you can? – Peter M. Mar 12 '18 at 14:06
  • David: If this question gets closed, comment here mentionig my name ( @PeterMasiar ) and I will vote to reopen it. Perfectly valid question, but is shows how many of participants of this forum have not enough experience and feel the urge to close questions not relevant to their narrow experience. Sad! – Peter M. Mar 12 '18 at 14:11
  • Completely agree, @PeterMasiar - this stack closes off questions way too quickly. It's destructive, unwelcoming and unnecessary. Can we start helping people out with their questions so we can provide answers, rather than voting to close everything? If a question genuinely needs closing, then please provide a reason why. – trashpanda Mar 12 '18 at 17:16
  • @theonlydanever - we have an ongoing problem with closing mafia. They gained enough reputation to close, so they do. Another ongoing problem is low-quality questions, and closing mafia lacks the experience to tell the difference. They just close all questions which they do not understand, due to their lack of experience. Wisdom of the crowds is madness of the mob in this sad situation, and I am not sure how to solve it, except removing rights to close from some people. – Peter M. Mar 12 '18 at 17:45
3

This may sound odd and may not be the answer you accept but I want to pass on my advice from the last 5 years of doing Selenium automation in several different companies from small local shops to national players. One of the biggest lessons I have learned from all these encounters is, surprisingly enough:

Avoid test data management at all costs.

Let me explain: Obviously you need some test data. Something has to be typed in all those form fields after all! What I have found is the pattern goes something like this: You start with a simple text name like 'TestLast'. Then you start extracting that into a separate data store (always with the 'but I am using it in multiple places' comment). After a while the test data starts to take on a life of its own. Worst of all the company starts asking for different scenarios and you start implementing different data scenarios for them. This starts to become complex very quickly.

So, my advice is avoid this journey. When you need a last name, hard code 'LastName' as the text. Doing it in multiple places is ok. Checking the value with a second hard-coded static value is ok. gasp! After all even if you use a variable of some sort you'll use that variable in every place. Yes having the variable in one place seems DRY but actually I have found provides little real benefit in comparison to the overhead that starts to get introduced with the dreaded "Test Data Management" issue. I've been there, done that and not repeating my previous path of complex test data is my advice now.

The reality of needing specific data to generate and test specific results is most frequently actually a code smell - you are using the front end to test logic that is usually implemented on the back end. Instead of UI tests you should have unit and integrated tests for that.

  • But what if you need it? If I have RegisterThing() method that is used in 10 tests, it would be silly to hard core values. – George Mar 12 '18 at 17:22
  • Exceptions can also be made as appropriate. My point is to avoid the usual path for this when possible. – Michael Durrant Mar 13 '18 at 0:20
  • In some applications you must have some test data that you cannot simply generate - especially if your SUT works with binary files. – ne2dmar Mar 14 '18 at 9:44
  • When you must have it you do. The problem is people aren't used to trying to avoid it generally. – Michael Durrant Mar 14 '18 at 10:06
2

I agree with @MichaelDurrant with one exception: instead of hard-coding all constants as literals where used, define bunch of constants in each test.

Reasoning: if something changes, most likely you will need to change both code and test values, so best is to have them in the same file.

Version control for the code will conveniently carry also versions of your test data, relevant to the version of the code.

1

I would recommend to design and baseline QA database copy which can be refreshed/restored before every regression run.

This helps to keep data in known base state before any run.

This strategy worked well for me in multiple large automation projects.

Up to large extent I would agree with @MichaelDurrant and verify maximum no.of data exhaustive edge cases on lower levels of test pyramid.

0

you might thought of using data builder pattern for test data generation. or you can also design an algorithm to extract data(masked) from production server to fit the data providers for your tests.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.