14

Meta: this is probably not a question for which there is a single, objective, best answer, but I'm going to ask it here anyway because it's applicable to our jobs and non-obvious.

I spent three months last year performance-testing a system. I thought of the process as a series of experiments of the form, "Configure the system like this, using that version of the code, and run the test using these variables." I kept notes on each experiment, e.g. the date & time, software version, inputs, resulting graphs. My notes were in a Google Doc that I shared with my co-workers. After the project was over, one of my co-workers complained that it was often hard to determine which experiments I had performed. In retrospect, I could have spent more time editing my notes to make them clearer. That I didn't wasn't just a matter of laziness.

Here's what I mean. In a scientific investigation, where you formulate a hypotheses and decide on how to evaluate your data before you run your experiments, it is easy to imagine recording all your experiments in a tabular form, e.g. a row for each experiment, a column for each variable, and a column for each measurement.

Performance testing, and other kinds of software testing, is more of a discovery process than a scientific investigation. You make mistakes, encounter dead ends, and discover variables that you did not anticipate in the beginning. In a sense, the discovery process looks more like a directed graph than a table. The linear flow of a table hides the hierarchical nature of the process.

So here's the question. What is an effective way to document the chronology of a discovery process? Are there tools you have used to simplify producing the documentation?

  • I think we have different definitions of "performance-testing", yours seems like "performance-discovery" – Rsf Mar 11 '16 at 9:51
  • This is about using performance tests as a tool to diagnose slowness. It's about more than just running JMeter and recording the results. You might try different server configuration parameters, or different kinds of inputs. Think about what happens in a Jira ticket for a complicated bug that requires a lot of experimentation. – user246 Mar 11 '16 at 16:04
  • Actually HTML started out as a means to organize scientific notes. For business purposes, best just stick to the end result, the stuff that you will be using after all discoveries have been incorporated. – Bookeater Mar 12 '16 at 14:43
  • 1
    I am actually brainstorming plans to build software that would assist with this exact problem. – Paul Muir Apr 16 '16 at 17:48
  • 1
    I would call what you are doing optimisation. Testing is measuring against something, a expectation or a baseline, performance testing would be the part you excluded, designing reusable tests, automating them and gathering the metrics they produce. – Martin Spamer Feb 18 '17 at 21:51
1

Update:

My answer today (months later) is to first question why? To what end? Yes I see that someone said your notes were hard to read. But trying to document the 'discovery process' would not seem to be to be an easy to do or understand task. I'm not sure of that would give the benefit to the other reader that you desire.
I would focus on displaying the final results in the tabular and chart form you describe using more concrete measures such as ones I list below.

Previous:
When considering performance I do not consider it a uncharted discovery journey.

I think key factors to guide performance testing should be:

  • What the response time standards are for the application ?
  • What are the requirements for wireless performance?
  • How does performance relate to key metrics, such as profit and enrollment?

Visualization can be be Jira tickets, Charts, presentations, performance tests locally and in CI/CD/Staging environments that are configured to accurately represent production machines and usage loads.

One option to consider is AB testing to learn how performance levels affect users.

  • Thank you for that but it doesn't address the question. I'm not asking how to performance-test. Finding performance problems is not a linear process. You explore options, hit dead-ends, and even re-run tests as you hone in on specific causes of bottlenecks. When you're done, it may not be enough to say, "Do this and you'll be ok." You may want to document how you reached your conclusions. The question is about ways to document that process. – user246 Nov 20 '16 at 17:24
0

One of the most informative mechanisms it to use graphical rather than tabular representations, these can be of various formats and indeed including multiple formats are often useful, e.g. a Venn diagram of the possible factors coloured for the number of tests passing can sometimes highlight combinations of factors that you had not though of. Likewise a graph of the number of tests and number of passing tests against a timeline, with a key for the test setup, can be very powerful as it is immediately clear that before a certain date testing of a given area wasn't done/possible. Some 3-D diagram types can be ideal as well.

Ideally you should not use a spreadsheet for this as you will be constantly updating it manually. Personally I like the combination of using tools like Jenkins for continuous integration & testing, version control with SVN/Git/Hg for the tests as well as for the code, test scripting, etc. You can use tools like python and tags in your test scripts and the results output to generate summaries like number of tests against each tag & results of tests against each result and save them to date indexed file(s), some test tools allow you to define meta fields that you can use for this, others allow comments that you can store/parse the tags in if you are using tool(s) that don't allow either there should be a test identifier that you can use index into a tags table if you are using test tools that don't support any of these then I would suggest that you stop using that tool ASAP as it lacks any possibility of traceability.

Once you have your data you can use tools like matplotlib, bokeh, plot.ly to generate the diagrams automatically for your daily/weekly/etc. reports - if you are using CI you can even have the charts updated on a per commit basis.

This paper also provides some interesting reading.

0

After having transformed our project's testing approach to a form that I haven't seen anywhere else but I find more meaningful, I have discovered, to my delight that it aligned perfectly with the "requirements" of the application.

I shouldn't go into details of that form, but my findings seem like an answer to your endeavor. First of all, I started calling every "verification" step of my test cases (regardless of them being exploratory or not) an "action-reaction" pair. The next step was to naturally define every verification as a "performance-target", i.e. a performance requirement.

I now see every requirement as a performance requirement, which we can safely call plain "requirement". And the next step is to assign a performance measure or group to "every" requirement.

Now if I come back to your question, I think the "exploratory" or the directed graph structure of your testing stems from the requirements that haven't been identified yet (which is more common than many may think, or actually is the natural way of design process).

With the last assumption, I should recommend you to just go back to a tabular report (or a hierarchical, depending on the structure of your requirements), and add any findings of your exploratory tests as new items to that collection. The need to define the graph-structure of your exploration is of course real, but is an unrelated phenomenon to identifying the "quality" of a version of the product.

0

So, the way I'd probably look at this, if I understand what you were doing:

When you start, you have a huge universe of possibilities. You've got different types of hardware you can use, different levels of software you can use, maybe different backends, maybe different data access patterns, etc. And what you're doing with your experiments is narrowing down the possibilities, until you find one, or a few, that are "best," based on whatever definition(s) of best you have.

So I'd present it that way. I had this n-dimensional space to start with. And with each experiment, I narrowed the space down in some way, or, possibly, expanded it some way.

There's something called simulated annealing you might look at, as it's trying to do the same thing, which is find the global optimum across a large number of possibilities.

  • Thank you for offering advice on how to approach finding a global optimum across an n-dimensional space. The question was about how to document an investigative process. – user246 Mar 20 '17 at 22:21
  • And that's what I'm saying you're doing. – Kevin McKenzie Mar 20 '17 at 22:32
0

The truth is (most)people don't like reading long reports. All they want is an answer to a question. So be brief, focus on presenting the conclusions and implications of the data and not the actual data. What did you do, how did you do it, conclusions on the data gathered(i.e. the answer), risks, unanswered questions(more things that need answers that the project cares about), the most important errors.

You don't give any details like project structure, who are you reporting to, do you have any helpers, can you yourself decide what is a good or bad value, and i think knowing these things might or might not help answer your question in a better way.

  • I asked the question because people wanted details. – user246 Apr 2 '17 at 15:02
  • Usually you collect the data and then you fix it up so it can be presented and you want a tool or method that means entering the data equals it being presentable immediately or at least quickly? – Hermes Apr 2 '17 at 16:32

Not the answer you're looking for? Browse other questions tagged or ask your own question.