Where do Fitnesse fits between unittests and API testing?

Question

Our team writes unittests for our python projects. Besides that, we have a tool that allows us to run scripted API tests (we can make requests and evaluate the result, passing or failing the test).

I am looking for a tool that helps the team to write acceptance tests for every ticket we do (primary objective), and then store these tests to start our regression test repository (secondary objective).

The problem is that not every ticket we do is API test-able, and sometimes I feel unittests are not enough (sometimes we need grey or black box tests). Some of our services don't have public APIs and they interact with other services in a lower level.

For example, we could have a ticket to change an internal function, but this should not affect our API response (we have tests to check this). Unittests could supposedly test this specific function, but this function can be called in other parts of our code (and even in other projects), so there's no way to write an unittest to check if this change won't break stuff across all of our services.

I was taking a look at Fitnesse and I was wondering: can Fitnesse fill in the gap between unittests and REST API testing, with the so called "acceptance tests"? What type of input/output can Fitnesse take?

Answer 1

Full disclosure: we rely heavily on FitNesse, run many hours worth of FitNesse tests daily in our CI buildbot daily, and I personally wrote both FitNesse tests (using Fit wiki format in browser) and WebDriver tests in Python.

FitNesse is for end-to-end testing (browser automation) and Selenium/WebDriver in much superior replacement, especially if you already use Python.

FitNesse failed and I would NOT recommend anyone to start with FitNesse today.

You are much better off using Python and call Selenium WebDriver directly. Debugging test cases written in FitNesse pseudo-wiki format is nightmare. Idea was that end users (non-programmers) would write tests in wiki format. It was not to be. Programmers prefer real language and real editor, and can by order of magnitude more productive. Just say No.

Jenkins is much better test runner. And if your pages use Angular, you need more granular control of what is going on on the page.

Also, consider PageObject design pattern. Easy to use in any decent real programming language. Not so much in FitNesse: In FitNesse, you either use "copy-paste inheritance" or write your own fixtures for every page.

Trust me on this one. We did started with FitNesse a decade ago (when it was state-of-the-art in system-level testing) but now we struggle to get rid of it: we need to gradually replace all those FitNesse tests and refactor them to PageObject pattern. It was good decision decade ago, but today? No.

Also read: http://www.jamesshore.com/Blog/Five-Ways-to-Misuse-Fit.html

Answer 2

I see two questions here:

• how to test something that is bigger than unit and possibly brings some business value to accept
• where does FitNesse fit here

To start with easiest, I support @peter-masiar in a way that it's better to stay away from FitNesse nowadays. It was a good, maybe break-though idea in its time, but the tool itself is utterly outdated and brings more problems than solves.

When you go to higher level testing, you might find the traditional "unit"-focused testing framework somewhat low level and not expressive enough. You might take a look at BDD or similar approaches and either implement a testing framework yourself or use some existing library. In java there are tools like cucumber and spock that fit this niche. If you prefer to stay with python then there seem to be some options either.

As for acceptance-testing the tickets that do not affect API, the question is what's the nature of those changes. If there is nothing to test via APIs that means that the change is non-functional. It can be either something you can still observer and maybe test indirectly (say performance improvement) or some kind of refactoring targeting improving of the maintainability of the code, in which case the only thing you would probably be concerned from the acceptance point is regression.

Answer 3

Results may vary... I've managed to create user acceptance tests for flight simulators with FitNesse (real stuff, not simulator games type).

Tests were written in natural language, convenient for the pilots and engineers, the calls to the simulation were wrapped into a small layer "stimulating" the switches and reading the values from the "cockpit" like airspeed, landing gear light status and many more.

Example of fixture in the wiki:

|Check|Landing gear light |NOSE|is|GREEN|

|Check|Landing gear light |LEFT|is|GREEN|

|Check|Landing gear light |RIGHT|is|GREEN|

|Move landing gear selector to |UP| position|

|Check|Landing gear light |NOSE|is|RED|

|Check|Landing gear light |LEFT|is|RED|

|Check|Landing gear light |RIGHT|is|RED|

example of code in C#

string LandingGearLightIs(string sIndic) {
    and so on ... return ("RED") or return ("GREEN")

Typical panel of the Boeing 737 for example: http://www.fly737ng.com/?p=228

The developers were writing tests the same way, that was very convenient because of natural language, learning curve was shorter when swapping developers.

Our framework contains about 37000 checks, spread in 900 tests about.