Automated software for data set management

Question

My team creates some kind of machine learning algorithm. We have an executable module, which gets input and produces output as files in some format. Thus each of our tests is such file. After module is executed we can analyze a result with other tool and get some parameters. Our goal is to maximize some output parameters.

Before we had all test files (one file per test) kept in one place, in a folder and it worked well just with running the script calling algorithm module for each file and gathering the results in a single file.

Now we have a need to attach some metadata for tests, for example, if some case is problematic, we could add a tag signalizing a type of problem in there. Or we could also attach case source tags. Also it would be great to keep history of parameters values.

So my question: is there a software which allows to manage not unit tests or tests based on source code, but tests based on data sets? I understand, that may be such software must be configured by creation of test sources and output-data adapters, but right now I cannot find anything which could do such job at all except TestComplete which costs. It would be great if this software have a web-interface.

Thanks in advance for any hints!

Answer 1

Software tools available for any task are very dependent on the language you use. You did not mentioned yours (my language of choice and reference is Python, one of the reasons is because it has such great tools and community). But most common languages do have most common tools, and all of the good ones are open-source and you can adapt/enhance them for your own usage, so here is idea:

I was looking at python-coverage tool. They have runmode when they measure coverage of code by unit test. Then, after a change in code, they can analyze the diff and re-run just unit tests executing the changed code. So you can adapt similar tool in your preferred language for similar use. Or measure which data sets exercise which code. or whatever you need.

You must realize that your use case is extremely specific and unlikely you will find some tool which does what you want "out-of-the-box". You will need a lot of customizations for any tool which will get you half-way to your goal.

That's why I like to use open-source.

Answer 2

For the time being Allure Framework is probably the most advanced reporting tool for testing. It has adaptors for many languages. It allows you to:

• Create steps and sub-steps in your tests. Sub-steps are good for detailed logging.
• Attach files like text files, JSON, XML, images, etc. You can view them right from the report in the steps. I think this is the metadata attachments that you seek for.
• Add marks like @Issue, @Story, @Feature that are going to be shown in different places in the report. It's possible to group by Features and Stories (Behaviour tab).
• Create test suites that's a different way of grouping of tests.
• It has a plugin architecture, so I suppose it's possible to add your plugins that will generate your own reports. But I haven't seen that done in practice.
• It also works with parameterization, but a test-framework needs to support this (e.g. TestNG has @DataProviders). Though for big string parameters or when there are many params it's better to use attachments.
• You can add descriptions for the tests as some kind of sub-title where you could add extra explanations.

You can either use a test framework for this or you could write you own adaptor which is not complicated - you'll need to use Allure classes/files to createStep, createSuite and so forth. Though for different languages it'll be a bit different. It'll generate XML for you and then will transform that XML into HTML report (via Maven, CLI, CI, etc).