API Testing with Specflow BDD Selenium?

Question

I'm developing a BDD test automation solution with C# using Specflow (Cucumber) with tests written as plain text Gherkins. I am currently writing a regression test suite amongst others.

When asserting that a user UI transaction has updated successfully i can think of a couple of approaches:

1. Using the Steps/Step Definitions: enter the data into the webpage and click save button, then assert that the success dialog box appears. Then navigate to another webpage or what could be multiple other webpages where the data will be changed and assert the data has been updated correctly in the necessary fields.

2. Using the Steps/Step Definitions: enter the data into the webpage and click save, assert that the success dialog box appears. Then write steps/step definitions that call API classes to retrieve the data stored (get) and then assert all the data stored to the database is correct.

The first option looks like a true representation of a functional test how a user would do it and it also guarantees the data is showing correctly to the user. The second option when I write Gherkin steps "Then the API call will return the following data:" - seems not right for a business readable format? There is also the chance that the data still may not be presented correctly on the UI. It would entail writing a number of API classes.

As a software engineering approach, is the second option a recommended approach?

Answer 1

It all depends on the logic you intend on testing, and when the test fails, how large an area of the application you'll need to investigate.

A full end-to-end test verifies data on screen. Doing this in BDD certainly replicates that test, but the behavior is what BDD is about. What behavior are you testing? How many reasons does your test have to fail?

If you replicate the full end-to-end test in BDD, and it fails because something does not appear on screen, you need to investigate quite a bit of code for the failure. If the test creates some data via the user interface, then verifies that data appears on screen on a different page, then what part failed? Did the "create data" page fail? Did the "view data" page fail? Did the call to a web API fail? Did the web API itself fail?

It becomes frustrating to debug this failing test, because you have so much to look at. Despite the frustrations a full end-to-end test is useful. It ensures all the parts work together. You do not necessarily need to add data to the system using the UI. It is appropriate to make direct database calls or web API calls from your Given steps. This is nice, because the tests will likely run faster, and the tests are less likely to fail on a Given step. Ideally you want a test to fail on a Then or When.

Scenario: ...
    Given thing A exists            # Calls database
    And another thing has been done # Calls web API
    When I do the thing             # Selenium interacts with the browser
    Then something should change    # Selenium verifies info on screen

Full end-to-end tests should be much fewer in number than the more fine grained tests asserting certain variations of behavior. Those require their own tests.

If you intend on testing the UI layer, then a full end-to-end test is appropriate. If you want to test the APIs themselves, I would not involve the user interface at all. Have your cucumber steps call the APIs directly.

Answer 2

In this case , The only need for testing the backend database is when you doubt whether the browser is actually getting the data from the database or is it simple caching it in cache or browser local storage.

So, if you want to ensure that's not what is happening, then use driver.quit() and open a new driver instance. This will be a new instance with all the local storage, cache etc are cleared. So you can be pretty sure that the data is indeed from backend and not a browser cache.

But if you still feel pessimistic that the site is not calling actual database and then, write a function to validate the backend and you can mask the API call in the spec flow step.

Eg:

Given("User sends data")
public void step(){
{
senddata(a)
verifydata(a) // step to verify api 
}

In BDD you don't have to show the implementation