3

Our team writes unittests for our python projects. Besides that, we have a tool that allows us to run scripted API tests (we can make requests and evaluate the result, passing or failing the test).

I am looking for a tool that helps the team to write acceptance tests for every ticket we do (primary objective), and then store these tests to start our regression test repository (secondary objective).

The problem is that not every ticket we do is API test-able, and sometimes I feel unittests are not enough (sometimes we need grey or black box tests). Some of our services don't have public APIs and they interact with other services in a lower level.

For example, we could have a ticket to change an internal function, but this should not affect our API response (we have tests to check this). Unittests could supposedly test this specific function, but this function can be called in other parts of our code (and even in other projects), so there's no way to write an unittest to check if this change won't break stuff across all of our services.

I was taking a look at Fitnesse and I was wondering: can Fitnesse fill in the gap between unittests and REST API testing, with the so called "acceptance tests"? What type of input/output can Fitnesse take?

7

Full disclosure: we rely heavily on FitNesse, run many hours worth of FitNesse tests daily in our CI buildbot daily, and I personally wrote both FitNesse tests (using Fit wiki format in browser) and WebDriver tests in Python.

FitNesse is for end-to-end testing (browser automation) and Selenium/WebDriver in much superior replacement, especially if you already use Python.

FitNesse failed and I would NOT recommend anyone to start with FitNesse today.

You are much better off using Python and call Selenium WebDriver directly. Debugging test cases written in FitNesse pseudo-wiki format is nightmare. Idea was that end users (non-programmers) would write tests in wiki format. It was not to be. Programmers prefer real language and real editor, and can by order of magnitude more productive. Just say No.

Jenkins is much better test runner. And if your pages use Angular, you need more granular control of what is going on on the page.

Also, consider PageObject design pattern. Easy to use in any decent real programming language. Not so much in FitNesse: In FitNesse, you either use "copy-paste inheritance" or write your own fixtures for every page.

Trust me on this one. We did started with FitNesse a decade ago (when it was state-of-the-art in system-level testing) but now we struggle to get rid of it: we need to gradually replace all those FitNesse tests and refactor them to PageObject pattern. It was good decision decade ago, but today? No.

Also read: http://www.jamesshore.com/Blog/Five-Ways-to-Misuse-Fit.html

  • Agreed 10 times – dzieciou Oct 3 '17 at 12:53
  • Interesting read! – FDM Oct 3 '17 at 14:43
  • 1
    Reports from the front lines are like that. :-) Hard part is: all this is experience hard earned by blood and sweat. – Peter M. Oct 3 '17 at 18:10
  • Peter, but then isn't this true for all BDD frameworks including cucumber? – Vishal Aggarwal Jan 8 '18 at 9:42
  • @VishalAggarwal - You are quite correct. Language designers are trying to create something "user friendly" and "in plain English" since COBOL (1959). So far, every time they failed. When I read "English is enough" as design principle of Cucumber, I knew it will fail, like all preceding attempts. English is NOT a programming language. In my experience, Python gets closest to "programming in plain English". I want my programming language I spent years to master, I do not want to work around features missing in Cucumber because Cucumber designers thought I would not need them. – Peter M. Jan 8 '18 at 14:38
1

Results may vary... I've managed to create user acceptance tests for flight simulators with FitNesse (real stuff, not simulator games type).

Tests were written in natural language, convenient for the pilots and engineers, the calls to the simulation were wrapped into a small layer "stimulating" the switches and reading the values from the "cockpit" like airspeed, landing gear light status and many more.

Example of fixture in the wiki:

|Check|Landing gear light |NOSE|is|GREEN|

|Check|Landing gear light |LEFT|is|GREEN|

|Check|Landing gear light |RIGHT|is|GREEN|

|Move landing gear selector to |UP| position|

|Check|Landing gear light |NOSE|is|RED|

|Check|Landing gear light |LEFT|is|RED|

|Check|Landing gear light |RIGHT|is|RED|

example of code in C#

string LandingGearLightIs(string sIndic) {
    and so on ... return ("RED") or return ("GREEN")

Typical panel of the Boeing 737 for example: http://www.fly737ng.com/?p=228

The developers were writing tests the same way, that was very convenient because of natural language, learning curve was shorter when swapping developers.

Our framework contains about 37000 checks, spread in 900 tests about.

  • Impressive. You wrote Fitnesse tests yourself and used FitNesse wiki/tables format just as a convenience for subject matter experts (to be able to see the inputs and outputs), not as "magic pixie dust" allowing users themselves to write the tests (because they rarely do). Yes, in such situation FitNesse wiki format is easier to read for subject matter experts than Python code, even if it adds additional effort to programmers. – Peter M. Jun 1 '18 at 20:28
1

I see two questions here:

  • how to test something that is bigger than unit and possibly brings some business value to accept
  • where does FitNesse fit here

To start with easiest, I support @peter-masiar in a way that it's better to stay away from FitNesse nowadays. It was a good, maybe break-though idea in its time, but the tool itself is utterly outdated and brings more problems than solves.

When you go to higher level testing, you might find the traditional "unit"-focused testing framework somewhat low level and not expressive enough. You might take a look at BDD or similar approaches and either implement a testing framework yourself or use some existing library. In java there are tools like cucumber and spock that fit this niche. If you prefer to stay with python then there seem to be some options either.

As for acceptance-testing the tickets that do not affect API, the question is what's the nature of those changes. If there is nothing to test via APIs that means that the change is non-functional. It can be either something you can still observer and maybe test indirectly (say performance improvement) or some kind of refactoring targeting improving of the maintainability of the code, in which case the only thing you would probably be concerned from the acceptance point is regression.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.