Using BDD to describe complex interactions in a state machine

Question

How would you use BDD to describe complex interactions with a state machine?

For example consider this scenario:

1. User enters input into the system.
2. The system transitions into state A.
3. The user enters more input into the system.
4. The system transitions into state B.
5. The user enters some more input into the system.
6. The system transitions into state C.
7. Verify that the system is indeed in state C.

Is this a single scenario? Do you describe steps 1 and 3 as Given, step 5 as When and step 7 as Then? What happens if steps 2 or 4 fail?

Answer 1

This is not a complex scenario, instead, it is common in many applications. Every application in a given user journey, goes through different states and at any step application can get into an undesired state and test might be failed.

So yes, it can be represented as an acceptance test/end to end test where on failing at any step, remaining steps will be skipped and the test will be marked as failed.

I think as a general rule of thumb you should try to minimize the number of asserts per test. However, as long as the test sufficiently narrows the problem to a specific place in the code, place it.

For error-handling, at any point in any step, if a step fails like 'Element not found' which raises an exception from selenium, which is handled at cucumber level which further fails the test. So as an automation developer we don't have to handle it explicitly.

As far as using given, when , then keywords are concerned , they are just syntactical sugar and under the hood they all are same as technically there is no difference. They are there to make a test more readable by nontechnical people as well.

Answer 2

Since BDD is about examples I find it only appropriate that this question be answered using an example of a state machine.

Suppose your business needs software for an oven and we have a coil that heats up to infinity. We need a way to control the temperature.

Through conversations with our oven experts we decide to define an acceptable range of temperatures the oven may be within when the user sets a temperature:

Feature: oven thermostat
    In order to heat up food
    As a user
    I want to be able to control the oven temperature 

    Background: 
        Given I set the oven to 350
        And the oven's acceptable minimum heat is 2 degrees less than the setting
        And the oven's acceptable maximum heat is 2 degrees greater than the setting

    Scenario: I turn the oven on
        Given the oven is off
        When I turn on the oven
        Then the oven will be on
        And the oven will check the temperature

    Scenario: the oven gets too cold
        Given the oven is at 0 degrees
        And the oven is on
        When the oven checks the temperature
        Then the oven will start heating

    Scenario: the oven gets too hot
        Given the oven is at 360
        And the oven is heating
        When the oven checks the temperature
        Then the oven will become idle

The point is, we describe the behavior of getting the oven between a range of temperatures. The state machine aspect is more of an afterthought.

As far as exceptions go:

    Scenario: the oven tries to heat but is already too hot 
        Given the oven is 500
        When the oven tries to heat
        Then the oven will not be able to heat

Answer 3

Looks like you are asking 3 questions instead 1 which makes the question not very clear, but I'll try to answer.

First question: nobody can answer this question without clarifications for the context of test,states details and states complexity. BDD approach is trying to define the real user behaviour. "State machine" is not a how the user sees the SUT. In BDD users point of view matters most. Without concrete context you can't do BDD tests. Using only gherkin syntaxes doesn't make your tests BDD.

Second question: No

Third question: I guess that only the creator of the test case can answer that question.

UPDATE:

If there is no other way to get to state C without passing through State A and B then yes single scenario. If the State C is the main state that you need to test then do less assertion (but not without any assertion) on steps prior step 6. This will assure if the there is failure in 2 and 4 steps then the test the will show you where exactly is the problem.

Q2: GIVEN 0. "Regular user" When 1. AND 2. WHEN 3. AND 4. WHEN 5. THEN 6. No step 7. I'll do extended asserts on step 6. On the AND Steps I do less assertions if possible.

PS:

Recently I had to do a big refactoring for almost 200 tests. Some of them had step with too high abstraction. I lowered the abstraction to the appropriate level and splitted some tests to two and more test cases where was possible. This decreases the time to figure out where exactly the problem was when some tests fails and also decreases the test maintenance time.

Answer 4

How would you use BDD to describe complex interactions with a state machine?

In your example, the smallest testable interaction would be between the fewest system inputs that cause change to each measurable outcome state to be the sinuglar tests.

Why? If you take a step back and consider in the future when you have a larger automated test suite, having feedback that directly points out what business rule has changed, will help with direct bug fault finding with every breaking code change.

Specifically to your example, I'd suggest you split your tests into just isolating each input(s) to state transition. Subsequent state transitions should be set in such a way that they could run in parallel without dependancy of having to run though previous tests, potentially feeding in requisite previous state as apart of the test setup.

You want to avoid having to do several assertions because you can conflate reasons for failure with ambiguous behaviour possibilities which you'll have to debug to understand what chained state input combinations actually caused said failure.

Answer 5

The typical test for a state machine in Gherkin would be:

• Test 1:
  • Given being in State A
  • When action A>B is taken
  • Then switched to State B
• Test 2:
  • Given being in State A
  • When action A>D is taken
  • Then switched to State D

Because a state machine is a machine that changes the state on the condition. Depending on how complex a system is (your E2E tests might need a sequence of states to be covered), you might extend your test to look like

• Test E2E:
  • Given being in Step A
  • When action A>B is taken
  • And action B>C is taken
  • Then switched to State C

However this makes the step implementation not very obvious and decreases the comfort of reading and understanding such the test.

Answer 6

I try to stick with a bunch of best practices when writing my BDD scenario's described on this blog.

1. Write the given step in such a way it described what needs to be ready before you start the actual test
2. Try to avoid multiple 'Given' steps to keep it readable
3. Try to avoid many 'When' steps (multiple steps might mean you actually need multiple scenario's)
4. Try to avoid a first person style of writing, instead use the third person ("When I...." versus "When the user....")
5. Describe the function instead of every action ("When the user clicks on the logout button" versus "When the user logs out")
   • Example: It leaves more space for interpretation, so when you describe navigating to a page, on desktop this could be click where mobile needs a slide or a tab. The actual solution you solve with the code behind it, not in the scenario. I'll come back to this in a later blogpost about mobile test automation
6. Use variables instead of absolute values
   • Example: Imagine testing on different environments with different data and the following scenario ' When the user reopens ticket "15"' . Now on the other environment ticket "15" might not excist but it's now "16". So it's better to put this info in a config file and rewrite the scenario to something like 'When the user reopens "the closed ticket"'.
7. Scenario's should stay readable, try to avoid element id's or classes in the scenario's, instead describe what you see
8. Avoid Steps within steps. This actually goes a bit more towards the technical part instead of Gherkin only and steps that refer to other steps are allowed but not best practice. Instead build functions and re-use those in steps

Answer 7

If you interested in step C only, your system must be in state B before test run(throw database, manually data preparation....)

Test precondition:
   System has state B

Scenario: State is checked after data was entered

   When  Enter data 'value'
   Than  Check the system has state 'C'

or if your have several systems:

Test precondition:
   System1 has state 0
   System2 has state A
   System3 has state B

Scenario Outline: State is checked after data was entered

   When  Enter data '<value>' in system '<system>'
   Than  Check the system has state '<state>'

Example:
| system  | value  | state |
| System1 | value1 | A     |
| System2 | value2 | B     |
| System3 | value3 | C     |