-1

Answer posted at the end of this post

Disclaimer

This topic might violate the general rule about how to ask questions. But I'm very sure its answers will add great value and hope the question will therefore not be closed.

Question

What resources (ideally books, because they're consistent) are there explaining about advanced test automation practices - Something like follow-up guide on auto testing after one is familiar with the basic concepts (Like if you read a programming book and now need to learn about software architecture).

Example topics that should be covered

  • How do you express the "functional dependency order" of tests?
    I.e. test2 makes use of functionality test1 tests exhaustively. How do I express that in a test? Background: I'd If test1 and test2 both fail, it should be clear, that test1 is the one I should look for an error first.

  • How do I express priorities for tests?
    E.g. there is test_show_error_message and test_software_starts_up. test_software_starts_up is the the test with more priority and if anything happens should be looked at first. How do I express that?

  • How do I express that tests take only short time or a long time to test?

  • How do I express that tests have external dependencies?

  • Unit tests, Integration Tests, System Tests, ... Do I have a completely separated test system for each of these? If yes, How do I make sure to not forget to start up each of them after a change?

  • How should I implement the same test with many different inputs?
    E.g. I want to test an add(a,b) function: do I have many different tests with rather silly names like test_test_lower_bound and make it difficult to overview or do I use some kind of CSV table to feed in data? Pros? Cons?

  • How do I handle database test?
    Do I build up and destroy a database server everytime a test starts that needs the database? Do I use one huge test database containing all test data for all tests and use transactions and rollbacks? What I test code that finishes transaction? What I I'm working with a MySQL database?

  • How do I maintain all of the tests?

What I've found so far

The following resources explain about these advanced topics:

Background

I see many, many books explaining the basics and abstract concepts of (auto) testing (e.g. why should we automate?). However, I cannot find resources explaining how to handle/manage/structure tests.

Out of scope topics

The following are topics, that should not be covered, as they should already be familiar concepts to the reader.

  • What is testing?
  • Why is testing necessary?
  • Psychology of testing
  • Unit tests are not integration tests
  • The Software development lifecycle
  • White Box VS black box testing
  • Test categories

Answer

(Still people here feel that answering your own question is a bad idea, so I'll post my answer here:)

It seems like there really isn't any resource like this.

For the sake of reference: I'm trying to gather information in this repository.

11
  • 2
    great set of questions. However the format of this site is not suitable for bundled questions (i see at at least 8 here) and the voting process. For example if one answer addresses 2 points well and another addresses a different 2 points well, etc, etc. Also this much content is not a question it is an ask for help that would take a lot of time. Time is money for most folks so a full answer may not be valuable to them Mar 13 at 13:24
  • Your first question for example might be "How do you express the "functional dependency order" of tests? I.e. test2 makes use of functionality test1 tests exhaustively. How do I express that in a test? Background: I'd If test1 and test2 both fail, it should be clear, that test1 is the one I should look for an error first." and that is more than enough for one question. Mar 13 at 13:25
  • @MichaelDurrant: Thank you. Your comment expressed, that my actual concern was not stated clearly enough. Your perception of questions where actually just a bunch of examples (/ example topics). I tried to emphasize the actual concern through a heading.
    – DarkTrick
    Mar 13 at 13:29
  • 1
    hmmm, they are all great question in my opinion. Each is a separate topic that deserves much attention. The main thing I wish you to consider is how to get answers fro the people reading this, i.e. their viewpoint of what they can give, not your viewpoint of what you need. Focusing only on your needs and getting an answer in the format that you specify may not lead to the answers and help that you desire. Mar 13 at 13:33
  • Programmers break things down into the smallest possible things and using that approach here may lead to greater results. At the end of the day exhaustive and advanced is just not a question Mar 13 at 13:34

1 Answer 1

0

While I'm on the fence on closing vs keeping this questions open, I'll give it a go at a general answer.

First, writing test automation is writing software. No one should tell you otherwise. So any book, course, etc that has a focus on writing software, architecting software, design patterns, etc, is valid and valuable when writing software testing frameworks! You can take those same concepts developers use and apply them to the testing side.

For example, when writing Selenium frameworks, we often use POM - Page Object Model - which is a popular software design pattern. There is nothing inherent about this design pattern that says it's "for testing only". In fact, it's used by developers in the applications they create. You could also use MVC or MVVC design models when creating testing frameworks.

I cannot find resources explaining how to handle/manage/structure tests.

This is just code organization, which different design patterns attempt to explain. A clear rule of thumb is to "keep like things together." You can also just have a basic directory structure. Note, not all of these are needed; use what you need when you need it; this is also non-exhaustive.

  • /lib
  • /logs
  • /config
  • /reports
  • /helper
  • /fixtures
  • /factories
  • /specs (or tests)

Under /specs, you can have:

  • /unit
  • /integration
  • /e2e
  • folders by feature, ecommerce example here can be: login, header, footer, checkout, search, pdp, cart (very POM oriented structure)

Don't put all your tests in one file. Aim to write clean code that is also readable. I've seen spec files 1000+ lines long due to poor organizing. You can also separate out tests by tags in the spec file or even in separate files: positive, negative, sub-feature.

Examples here can be:

  • featureName.positive.spec
  • featureName.negative.spec
  • featureName.subfeature.positive.spec
  • featureName.subfeature.negative.spec

When it comes to the spec file, make sure that it's just about the test (assertion) and code that supports getting to that assertion. Any code you keep repeating is meant for a helper file, as a fixture, or as a Page Object (if you're doing UI tests). Keep your code DRY (don't repeat yourself) not WET (we enjoy typing)!

Unit tests, Integration Tests, System Tests, ... Do I have a completely separated test system for each of these? If yes, how do I make sure to not forget to start up each of them after a change?

Other ways to phrase it:

  • Do you keep all your tests in the same repository as the main application?
  • Do you only keep unit tests in the same repository as the main application?
  • Do you keep all your tests in a separate repository?

Yes, to all. No, to all. It really depends on team structure. Running them via separate repos can be handled via a CI/CD system.

How do you express the "functional dependency order" of tests?

You don't. Tests should follow the FIRST principle. Plenty has been written about this including me on this answer. In general, tests should follow an order, they should be independent of that.

How do I express priorities for tests?

A lot of test runners and test libraries have a tagging system. You can use that to target tests by tag, e.g. smoke tests, feature name, security test, etc

How do I express that tests take only short time or a long time to test?

Not sure there is an answer here. Is this question about writing tests? Running the tests? There is a different answer depending on which task.

How do I express that tests have external dependencies?

You don't. It's inherent to the type of testing. Unit tests usually don't have external dependencies. Integration tests usually do. Sometimes you can mock those external dependencies.

How should I implement the same test with many different inputs?

You can write one test that takes different values of inputs. Here, add() is the method you call in your spec file that should return a value that you assert against.

add(1, 2)
add(-1, -2)
add(0, 0)

Or, you create separate tests in your test library

it "adds 2 positive numbers"
it "adds 2 negative numbers"
it "adds 2 floating point numbers"

Or, you can store input values in a JSON object, array, CSV file, etc that just loops through those values in a single test.

Many options. It's dependent on context.

How do I handle database test? Do I build up and destroy a database server every time a test starts that needs the database? Do I use one huge test database containing all test data for all tests and use transactions and rollbacks? What I test code that finishes transaction? What if I'm working with a MySQL database?

You should ideally have a separate testing environment with a test database ready for use. The testing environment should mirror what is in production. On a team, this is usually already setup for you, usually by a DevOps team. For proper e2e testing or CRUD like test, yes, do clean up your test. All testing libraries have beforeAll, beforeEach, afterAll, afterEach to setup and clean up the tests.

How do I maintain all of the tests?

Like any other software. You will change the tests over time. You will delete things. You will add things. It gets maintained just like your main application.

Just to reiterate, most of what I'm describing here is NOT testing related. Its standard software architecture, design patterns, organizational principles, so any books and resources you find on those can be used when creating software tests, frameworks, infrastructure, etc.

Would it be useful for someone to write about all these things from a testing/QA perspective? 100%, but not entirely necessary.

3
  • 1
    Thank you for your effort on going over the questions. It doesn't solve the question, but it gives some interesting insights. I didn't expect anyone to go over the concrete questions.
    – DarkTrick
    Mar 17 at 0:26
  • @Use principles for structuring domain code for test code (1) Non-test source code already suffers the problem of only being able to structure after one principle (like structuring music within folders and then figuring out, that you don't know if you should structure by artist, genre, song name, song number, ...) (2) I'd say a test is like a tiny program. Many tests are like many tiny programs. The main application under test is just one program. So the problem at hand is already slightly different.
    – DarkTrick
    Mar 17 at 0:27
  • @DarkTrick Not really sure than what you're looking for as an answer. It seems like you might be stuck on code organization. Recommend reading up on Design Patterns. For your music example, structure it anyway that makes sense to you. Get practice with different organizational styles. It may take time, but you'll learn what works best and why.
    – Lee Jensen
    Mar 17 at 21:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.