Telemetry for feature health

Question

Not sure how opinion based my question is, but when you have a released (client facing) feature, how are you evaluating with telemetry if it's healthy? By healthy I mean users can interact with it (it's accessible) and when there was an interaction the expected outcome happened.

Here is an example for the expected outcome case: There is a delete button and if it's pressed, a request sent to a server, and if that returns 200, the delete functionality can be assumed to work.

Let's say we are sending a metric DeleteButtonPressed when the user presses the button and DeleteSuccessful upon receiving the 200 response code. If there is a drop in DeleteSuccessful/ DeleteButtonPressed it can be said that the feature is not healthy.

However how do we know if the feature is accessible? The problem is that it cannot really be distinguished if the users are not be able to use the feature (because of let's say in a specific OS version the app behaves differently and the feature is not visible) or they don't want to interact with it. Monitoring just the rate of DeleteButtonPressed therefore isn't a good indicator.

Maybe I'm missing something but what are good metrics to monitor feature health?

Answer 1

Summary: It's a reasonable concern but not one that I encounter in practice.

Create smoke tests that are high level are assert that basic functionality, such as visiting the first page, work. These should not be reliant on OS specific peculiarities about edge cases.

Run these tests as part of the deployment to an environment process and only consider the application to be deployed if they all pass.

Also, perform exploratory testing to ensure that the elements are truly visible.

For a specific feature as you mention, part of the puzzle is making sure you have quality unit tests, good integration tests and good Acceptance tests. Acceptance tests including UI tests can become part of your regression suite but this should happen selectively and mostly not happen otherwise you start building a massive and slow regression suite. Most business struggle with this because it seems attractive except that... it gets slower and slower and businesses today want speed.

I've been writing selenium tests for years and I have not experienced what you describe as a common issue actually occurring. I can certainly recall ONE TIME, where in IE you had to scroll down before the selenium finder would work - but even in this case, it worked for actual users, just not the automation without an additional scroll_to action.

Answer 2

Telemetry can't always pin point problems, but many times it can indicate the existence of a problem.

If you expect some problems to occur you can sometimes add smarter telemetry and better analysis of other pieces of information, for example users skipping the Delete button and closing the application all together.

A complementary approach is using A/B testing, give some users a Delete button of type A and some of type B and compare the results. You can use A/B testing to assess designs but also retrospectively to locate or fix problems.

Answer 3

To be able to effectively collect "telemetry" of a feature this capability should be initially (and intentionally) included into application architecture design. This is not that straightforward since it depends on many things like whether you need real-time monitoring, or, say, overnight analysis.

Generally speaking when you say that a feature is "alive" it usually means that is passes all the step sequence to deliver the result to the end-user. That result might be wrong one (however in my understanding that still means the feature is alive, but having a defect). To achieve that each high-level logical step composing a feature should log the step definition to the audit storage. Having such entries associated with user session and feature identifier you would be able to analyze if all the features produced the end result.

You would also need to define termination mark since sometimes the user just change their mind and do not complete the steps to accomplish the use-case.

So the metric could be the number of sequences which have no end-points achieved.