3

I've met with two approaches for test automation in the companies I've worked for:

  • Frameworks: providing unified format for test reports, same syntax for writing all tests, and other tools, maintained in one repo and delivered as one artifact.
  • Libraries, each library doing exactly one thing that teams or even single tester can pick up and combine together with other libraries. Libraries are maintained separately.

What factors would you consider when choosing between a framework and a set of libraries when building test automation for your company?

Here are my initial observations:

  • I've found frameworks to be good to enforce certain conventions: format for tests and test reports. It seemed to work better in the teams that test similar products or better, same product.

  • I've found libraries to be better for more diverse products and teams working in different ways, where one solution does not suit everyone.

  • Such teams needs to have more programming experience, because they need to pick up and combine libraries themselves

  • Teams that are using libraries are less that constrained than teams using a framework.

  • Finally, I've found libraries easier to maintain than frameworks.

Improve this question
3
  • 1
    IMHO, the question seems a little vague and abstract and bit more concrete contextual information will be helpful to get better answers.
    – Vishal Aggarwal
    Mar 9, 2019 at 23:30
  • @VishalAggarwal Which part is too abstract? To me this question is similar to sqa.stackexchange.com/questions/822/…. I'm not asking about a specific problem but rather about set of guidelines to decide in virtually any project.
    – dzieciou
    Mar 11, 2019 at 12:21
  • Without any contextual boundaries it is too broad to answer in any practically meaningful way.
    – Vishal Aggarwal
    Mar 12, 2019 at 1:24

3 Answers 3

Reset to default
2

This all comes from the demand that exists in your particular project. A library is just a way how you distribute your code. Frameworks can be distributed as a single library or set of libraries as well.

In my understanding a framework is the code that introduces inversion of control paradigm. So that you do not call a method a and then method b. You rather write a code that is called back by the framework. In this definition I would say frameworks are harder to maintain and require more coding skills from the maintainers.

The benefit is that frameworks will take care of execution context preparation so that the user will not care of how to set up everything. Rather how to write test well.

Improve this answer
5
  • To me a library is something that follows a Unix philoshophy of "Do One Thing and Do It Well.". Perhaps this was not clear from my question... So framework does not fit into that definition.
    – dzieciou
    Mar 11, 2019 at 12:28
  • Re: "This all comes from the demand that exists in your particular project. " By the demand do you mean only how the team wants the framework to be distributed? What exactly do you mean by demand?
    – dzieciou
    Mar 11, 2019 at 12:33
  • 1
    I mean what is the expected way you're code will be used in your project. Whether you plan to use it as a solid solution or you plan to integrate different parts (aka libraries) to different components and you need to maintain lightweightness of your build with tests. What are the restrictions from your sdlc processes and from your equipment, etc. This is what I mean.
    – Alexey R.
    Mar 11, 2019 at 12:43
  • Basically usually one starts implementing their framework because there is certain demand. Imho it is not quite effective to implement framework (or libraries) without clear demand since when you will run it in "battle mode" it might turn out that nobody considers it convenient.
    – Alexey R.
    Mar 11, 2019 at 12:46
  • Right. However, very often demand very clear in the beginning of test automation process. This is why often framework/libraries evolve as your knowledge about the product and the way it need to be tested evolves. It takes time to make your automation convenient. For instance, sometimes, you might think that the framework is the best for your team and then, suddently, you may learn that libraries are a better solution, like those guys: youtube.com/watch?v=PeioFobaq94.
    – dzieciou
    Mar 11, 2019 at 12:52
2

This is a very complex and opinionated subject. 'Framework' is also a vague term that means many different things to different people.

In the way you're splitting them out, I would say "Choose both". Where I work, we write libraries of common code (web testing code, app testing code, commonly useful code) and then use them in our frameworks (plural).

In the video you cite on Alexey's answer, the presenter is explaining how building a single framework and trying to make all of your projects fit into that one framework can be very difficult. It's probably not a great idea to do this, although I'm sure some people are good at it. It's not a negative mark against all frameworks though, they can be much smaller and more specific to a project.

I would suggest to use a framework per test project, backed by common code from your libraries. It shouldn't try to fit all of your test projects, just that specific one. Use the common code from your libraries to make things consistent and take away some of the boilerplate things you have to do in every single test no matter the framework, e.g. with Selenium-Webdriver we have a base test in a common library that decides what browser to start up. Replicating this code on a per framework basis would be wasteful.

Use your framework to make the library code easier to use in the specific context you need for the set of tests the framework is built to handle. Hide the complexity and make it easy for test writers to do what they need to do: write tests.

Improve this answer
1
  • Thanks for a pragmatic approach. It seems to combine advantages of both frameworks and libraries.
    – dzieciou
    Mar 11, 2019 at 19:21
1

The question seems bit abstract but in general I think the choice depends on how broad is the audience the framework/library is supposed to serve.

The broader the audience the more general in nature of the code will be and will tend more towards being libraries.

In one of my automation projects for an online shopping application multiple UI teams were using different UI automation frameworks implemented in different languages as well. One need arose is to design & maintain UI locators in an language agnostic way in one place which can be called from any framework/language and will be developed & maintained by a separate team , in the form of a general library.

And the reason was, as being UI based application locator maintenance itself was an substantial activity which was redundant between multiple project teams so it was abstracted out as separate library (in the form of XML) to cater multiple UI teams.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.