1

I've been tasked with finding a tool/solution to present our automated python test cases to other stakeholders in the business (I won't write actual python for time-saving purposes).

Our tests are structured as follows:

[Python test file]
User journey #1
- Code line 1 ("login")
- Code line 2 ("fill_in_the_welcome_form")
- Code line 3 ("verify_the_form_details_were_saved")

[Python DSL file]
Code line 1 ("login")
- Click "Login" button
- Enter Username value into username field
- Enter password into password field
- Click "Proceed"

Code line 2 ("fill_in_the_welcome_form")
- Fill in field 1
- Fill in field 2
- Fill in field 3
- Click "Save" button

Code line 3 ("verify_the_form_details_were_saved")
- Verify value "x" was saved into field 1
- Verify value "y" was saved into field 2
- etc
- etc

So we have reusable blocks of code in the DSL level that we can piece together like Lego in the test files and they are named in clear English sentences.

I can say to the stakeholders that "We have a test that fills in the Welcome page" but that's not helpful. Plus they can go to GitHub and look at the test files in our repository but that's fairly long-winded. Plus each line isn't very descriptive. "Fill in field 1" doesn't say anything about exactly what value is being inputted. The only way they could find this out is by going deeper into the code and checking what test data we provide.

We have tried documenting the test steps in Confluence/JIRA/Spreadsheets etc but it's a massive overhead to manually update all the documentation every time the product, and then test code; changes.

In your experiences, have you used a test management suite or tool where you can just document test steps? Preferably something that can be interacted with via an API, so the documentation can be easily updated from the code level. We use JIRA for everything, so any kind of plugin would be amazing.

Sorry for the long post but hopefully someone can advise. :)

Thank you, Jack

5
  • 2
    There's lots of information about WHAT you're doing, but not WHY you're doing it. Why the stakeholders want to see it in the first place? What will they do with it? "The only way they could find this out is by going deeper into the code and checking what test data we provide." perhaps they are interested in what test data you use? Is that the purpose?
    – pavelsaman
    Jun 24, 2021 at 21:00
  • Have you tried BDD with Gherkin syntax? Using tool like Cucumber or a Pyhton version of that?
    – Mate Mrše
    Jun 25, 2021 at 8:29
  • 1
    Thanks for replying @pavelsaman The stakeholders are interested to know what kind of coverage we have in our automated tests. They didn't want small functional tests, but instead larger/longer tests to mimic what a user might do in our software in order to provide regression testing and catch sev-1 issues. Jun 25, 2021 at 8:43
  • Hi @MateMrše Thanks for replying. Our tests aren't really on the small functional level like that. They are larger tests that mimic what a user would do in our software and so may span multiple areas of our software, instead of just one field for example. Jun 25, 2021 at 8:48
  • 1
    An End-to-end test is a functional test (like using Selenium to do some work flow in your webapp) and that is what BDD (Cucumber) is all about. Also, you cant get "code coverage" from UI tests, it doesn't work that way. Code coverage is typically calculated by a UNIT testing tool that is tests methods within the code directly.
    – kirbycope
    Jun 26, 2021 at 18:10

2 Answers 2

1

I think there are two parts to this.

  1. Write self-executing, living documentation through using BDD code as @kirbycope indicates.

  2. Figure out a way to educate non-technical folks that 100% coverage never happens at a functional level (don't call it an 'education' of course, call it an opportunity for a deeper conversation about value or some such bs). 100 coverage isn't usually practical once you consider the number of devices, browsers, versions, landscape mode, network speeds, etc. not to mention different data and the possibilities are essentially endless. So the focus needs to be on what is worth testing. And what is worth testing at the UI vs unit test level. How important is speed of code to production vs disruption to users from bugs that aren't fixed quickly vs how long the test suite is taking. Conversations and education about options and choices are the successful outcomes here.

2
  • Thanks for replying. We've tried a BDD approach in our code twice and it was rejected twice by the powers that be. For now, I think my only option is to type up as much test information as possible in JIRA tickets and then link them (by means of a code/label/marker etc) to the automated test. Jul 6, 2021 at 15:49
  • Sorry to hear that. The real world is tough. You may need to stick with polite but constant repetition. It can take many attempts. Be a little annoying but super polite. Aim to get buy-in before trying again. Trying to get people to change is very hard, the best approach is usually educate and then let them discover. Also study MBA, gartner, etc and use those terms in your descriptions. Talk business impact and value as much as possible. Jul 6, 2021 at 16:16
0

I use the "living documentation" and clear language of Cucumber. The Sceanrio Outlines will show what data is being as input. You can give everyone Read access to the repo (works for my smaller clients) or write a script to pull it into a spreadsheet (works for my government clients).

Scenario Outline: eating
  Given there are <start> cucumbers
  When I eat <eat> cucumbers
  Then I should have <left> cucumbers

  Examples:
    | start | eat | left |
    |    12 |   5 |    7 |
    |    20 |   5 |   15 |
1
  • Also, write Declarative steps and not implementation based tests. Because BDD tests are intended written before the implementation and can be written by the stakeholder during ticket grooming, a Dev, a BA, QA, etc.
    – kirbycope
    Jun 26, 2021 at 18:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.