1

Is there any tool for tracking testing pyramid state which visualises each stage of pyramid and where I can track disproportions of that?

p.s. I did a quick search and as I can see there are a plenty of tools for code coverage (this is mostly for unit tests) but nothing for tracking a whole pyramid.

Improve this question
6
  • 5
    The Test Pyramid is a heuristic - its goal is to be a general approach, not an objective/quantifiable metric. Such a tool would be misleading. A heuristic aims to guide, not to evaluate objectively.
    – João Farias
    Commented Sep 23, 2019 at 15:54
  • Test Pyramid tells us about proportions of tests on each layer and proportions can be metric and you can display it and track it.
    – Orkhan M.
    Commented Oct 10, 2019 at 10:33
  • Mike Cohn does not talk about number of checks anywhere in Succeeding with Agile - he talks about risk assessment, which by nature demands an heuristical approach. Better check your sources rather than rely on second-hand interpretations.
    – João Farias
    Commented Oct 10, 2019 at 11:02
  • Sorry, but that sounds very superficially. Tool I speak for is not the only thing you'll rely on - but in complex it could help to detect smth going wrong.
    – Orkhan M.
    Commented Oct 10, 2019 at 12:16
  • I just pointed out that such a tool would have only a weak relation to Cohn's Test Pyramid since this heuristic does not talk about what the tool does (counting number of checks). And "tracking testing pyramid state" implies a strong relation.
    – João Farias
    Commented Oct 10, 2019 at 12:53

1 Answer 1

Reset to default
3

I like the guidance from https://automationpanda.com/tag/testing-pyramid/: each layer of tests should be an order of magnitude smaller than below.

The Testing Pyramid should be a guideline, not a hard rule. Don’t require hard proportions for test counts at each layer. Why not? Arbitrary metrics cause bad practices: a team might skip valuable end-to-end tests or write needless unit tests just to hit numbers. W. Edwards Deming would shudder!

Instead, use loose proportions to foster better retrospectives. Are we covering too many input combos through the Web UI when they could be checked via service tests? Are there unit test coverage gaps? Do we have a pyramid, a diamond, a funnel, a cupcake, or some other wonky shape? Each layer’s test count should be roughly an order of magnitude smaller than the layer beneath it. Large Web apps often have 10K unit tests, 1K service tests, and a few hundred Web UI tests.

The problem generally is that Unit tests, Integration tests and UI tests are written in separate efforts. With different frameworks, languages, reporting, etc.

I think your two options are:

  • Use an complete testing solution vendor and get feeds from all the different testing areas to construct a visualization of your pyramid.

  • Manually have each layer communicated test totals to another small application that aggregates them for display.

Improve this answer
4
  • 1
    I've suggested a format improvement, but really only the last section (starting "The problem generally...") actually addresses what the OP asked.
    – jonrsharpe
    Commented Sep 23, 2019 at 16:09
  • Guidance is still guidance, that tool is in need of a team and team's product owner to track the state of pyramid - that helps to follow guidance. The difference of code-stacks on each layer of pyramid is not a real problem, you just need to have access to source code and know masks of each layer tests, then you can summarize them and bring a pyramid-diagram to life. Yes, to do that by my own is the option, but I'd like to know if there's any tool before I'l start implementing my own.
    – Orkhan M.
    Commented Oct 10, 2019 at 10:27
  • ^ a classic 'easy if you already know it' otherwise very hard. The underlying problem there is that we are not all-knowing
    – Michael Durrant
    Commented Oct 10, 2019 at 10:28
  • @MichaelDurrant I'm not sure I've got that you said
    – Orkhan M.
    Commented Oct 10, 2019 at 10:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.