I'm trying to rethink our test cycle that we use at my company and am wondering if anyone could give any leads, examples, experiences, etc. to how you test without the use of crowdsource data. The products we make is strictly for professionals whose time used on our products is limited and expensive, so a crowdsourced data model could cause a bad reputation like we've seen with our competitors.
Our tests require tons of manual testing in a variety of permutations for a single function, and I'd like to make things more efficient. Think of it like this:
main test 1:"test graphic functionality"
test 1: test on windows
- sub test 1a: test on 32-bit machine
- sub test 1b: test on 64-bit machine
test 2: test on mac
- sub test 2a: test on 32-bit machine
- sub sub-test 2ai: test on 32-bit machine using a "example add-on"
- sub test 2b: test on 64-bit machine
We currently use JIRA and I'm looking into Zephyr right now, but not liking it too much because of its lack of modularity (ie. once we come up with more products and as systems get more advanced, adding sub test cases become more difficult to manage)
Edited in response to answers:
thank you for the input Stacy and Kate!
I mulled over this a bit and thought on the other hand, it may be how we are organizing each project and how we are creating test cycles that could be the reason to my conundrum.
Currently for each 'app' we release, we support functionality on many different 'platforms' (like having support for 4 different OS's), and then for each OS, we allow support on 32 and 64 bit OS's.
We also port our 'app' to be run on 4 drastically different 'programs' on each OS (and each program has new versions coming out all the time).
And as if that wasn't enough, each 'app' can run at 4 different quality 'resolutions' (ie. standard def, high def, etc.).
All while expecting each permutation to have the same functionality and quality.
Unfortunately, it may be due to the high bar we've set for ourselves that we are stuck at having to test every single permutation individually to make sure one of those permutations doesn't make our release a stop-ship...And seemingly, there's always something going wrong with each release at some weird combination of the aforementioned (be it regression or otherwise), so it's hard to cut corners for any of these requirements.
And with the future fast approaching, I can only imagine more permutations emerge to the point where it becomes too much to handle. And that's where I'm really stuck at. So I'm trying to brainstorm ways to organize all of these releases into projects and test cycles and so on since this high standard we set for ourselves has made all these 'feature-like' combinations 'necessary' for our end users to be happy.
Thoughts on that?