My team creates some kind of machine learning algorithm. We have an executable module, which gets input and produces output as files in some format. Thus each of our tests is such file. After module is executed we can analyze a result with other tool and get some parameters. Our goal is to maximize some output parameters.

Before we had all test files (one file per test) kept in one place, in a folder and it worked well just with running the script calling algorithm module for each file and gathering the results in a single file.

Now we have a need to attach some metadata for tests, for example, if some case is problematic, we could add a tag signalizing a type of problem in there. Or we could also attach case source tags. Also it would be great to keep history of parameters values.

So my question: is there a software which allows to manage not unit tests or tests based on source code, but tests based on data sets? I understand, that may be such software must be configured by creation of test sources and output-data adapters, but right now I cannot find anything which could do such job at all except TestComplete which costs. It would be great if this software have a web-interface.

Thanks in advance for any hints!

  • If I understood properly, you are asking something like data parameterisation in test cases. Based on parameters you wanna perform different things. Am I right? ==> santoshsarmajv.blogspot.in/2013/02/… Nov 17, 2015 at 14:17
  • Yes. The parameter is a file name. We want to add tests and manage them just by adding a metadata.
    – Dmytro
    Nov 17, 2015 at 14:38
  • what do you mean by "manage them just by adding meta data" ? Nov 17, 2015 at 14:42
  • In your example that is xml file with suite parameters. For us it would be a semi-automated tool, which allows to add file - and automatically modifies this xml or whatever configures the suite.
    – Dmytro
    Nov 17, 2015 at 15:23
  • 1
    You can have your framework kinda thing for this. Instead of reading the data from file, make use some database to get the latest parameters. Once the file has been uploaded, update the details in DB which behave like your suite from which you are going to read params. Nov 17, 2015 at 16:11

2 Answers 2


Software tools available for any task are very dependent on the language you use. You did not mentioned yours (my language of choice and reference is Python, one of the reasons is because it has such great tools and community). But most common languages do have most common tools, and all of the good ones are open-source and you can adapt/enhance them for your own usage, so here is idea:

I was looking at python-coverage tool. They have runmode when they measure coverage of code by unit test. Then, after a change in code, they can analyze the diff and re-run just unit tests executing the changed code. So you can adapt similar tool in your preferred language for similar use. Or measure which data sets exercise which code. or whatever you need.

You must realize that your use case is extremely specific and unlikely you will find some tool which does what you want "out-of-the-box". You will need a lot of customizations for any tool which will get you half-way to your goal.

That's why I like to use open-source.

  • Python is good for us. We run our tests as an executable or script. And yes, seems we will need to create our own solution. What can you recomend for python?
    – Dmytro
    Nov 19, 2015 at 12:56

For the time being Allure Framework is probably the most advanced reporting tool for testing. It has adaptors for many languages. It allows you to:

  • Create steps and sub-steps in your tests. Sub-steps are good for detailed logging.
  • Attach files like text files, JSON, XML, images, etc. You can view them right from the report in the steps. I think this is the metadata attachments that you seek for.
  • Add marks like @Issue, @Story, @Feature that are going to be shown in different places in the report. It's possible to group by Features and Stories (Behaviour tab).
  • Create test suites that's a different way of grouping of tests.
  • It has a plugin architecture, so I suppose it's possible to add your plugins that will generate your own reports. But I haven't seen that done in practice.
  • It also works with parameterization, but a test-framework needs to support this (e.g. TestNG has @DataProviders). Though for big string parameters or when there are many params it's better to use attachments.
  • You can add descriptions for the tests as some kind of sub-title where you could add extra explanations.

You can either use a test framework for this or you could write you own adaptor which is not complicated - you'll need to use Allure classes/files to createStep, createSuite and so forth. Though for different languages it'll be a bit different. It'll generate XML for you and then will transform that XML into HTML report (via Maven, CLI, CI, etc).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.