My team creates some kind of machine learning algorithm. We have an executable module, which gets input and produces output as files in some format. Thus each of our tests is such file. After module is executed we can analyze a result with other tool and get some parameters. Our goal is to maximize some output parameters.
Before we had all test files (one file per test) kept in one place, in a folder and it worked well just with running the script calling algorithm module for each file and gathering the results in a single file.
Now we have a need to attach some metadata for tests, for example, if some case is problematic, we could add a tag signalizing a type of problem in there. Or we could also attach case source tags. Also it would be great to keep history of parameters values.
So my question: is there a software which allows to manage not unit tests or tests based on source code, but tests based on data sets? I understand, that may be such software must be configured by creation of test sources and output-data adapters, but right now I cannot find anything which could do such job at all except TestComplete which costs. It would be great if this software have a web-interface.
Thanks in advance for any hints!