5

We keep having the idea of logging user actions at a detailed enough level to do things like track feature usage, identify common workflow errors, etc.

If we had such detailed logging, it seems like it would be trivial to write something to convert those logs into automated tests.

Does anyone here have experience with trying this sort of thing, and if so, what were the challenges?

Assume using web technology and that the html itself is well-written for testing (well-defined locators, etc., but feel free to enumerate these better than I can--it's likely at least half the battle!). Obviously there's the risk of the generated scripts breaking if the UI is updated in a way that would break the locators. But they might provide a good starting place for editing to make the test more flexible.

This would also be extremely useful for reproducing reported bugs, in which case you could run the log-generated "test" on the exact version it was logged from, so UI changes wouldn't be an issue in that use case.

Improve this question

1 Answer 1

Reset to default
4

Record/replay might be valuable to record all steps necessary to make a bug reproducible. But what if running recorded sequence the next day you need to enter next day's date, and your page has no "today", but concrete date was entered? All your pre-recorded steps are useless now.

And for regression testing, you want to use PageObject design pattern, so business logic, locators and actions are centralized, so if you need to change them later, you don't have to change all instances of record/replay which use them.

I can see use-case for record/replay, when manual tester records performing the operations and later can review if interacted elements have good locators (easier that inspecting locator for every element being interacted with).

Other than that, record/replay is IMHO of limited use, and only for pages which are designed with testing using record/replay in mind ("click here to insert current date", etc).

Bit more about our experience.

We have a (huge) suite of tests similar to record/replay (FitNesse) and (slowly growing) suite of new test which use PageObject patern. FitNesse are fragile, and if GUI/business logic change is made, many test break and needs fixing in many places. As a result of this fragility, test developers are reluctant to stray away from "happy path", and most tests are repetitive. We are working on retiring the whole suite (and replacing them with PageObject tests).

If PagoObject tests break, they likely need fixing in a single place, so as a result we are more adventurous and more willing to implement functionality straying away from "happy path", inspecting data and selecting some data in random, creating more "fuzzy" tests - and it is also much more fun for test developers to develop these tests (because developer can use Python debugger to inspect located elements), and such tests are much easier to debug - especially later, in maintenance phase when something changed and test feailed.

So (in my experience), record/replay tests are of a limited use as a starting point for good PageObject-related tests. Hard part is to design good page methods for each page (useful for tests), separating the interaction in proper size "chunks" where group of widgets are acted upon in a single action which makes sense for a user/test. Getting locators and order of interaction (which is generated by record/replay) is the trivial part.

Improve this answer
5
  • In your particular example, you could easily edit the generated script to replace the hardcoded date with your language-of-choice's function for getting today's date. But yes, there's still extra work involved to do that.
    – c32hedge
    Commented Nov 2, 2017 at 18:57
  • @c32hedge - define "easily". Whole point was not editing/maintaining the recorded script, IIUC? Also, date was the trivial example. I have scripts which select one of the random objects of different types for different users.
    – Peter M. - stands for Monica
    Commented Nov 2, 2017 at 19:32
  • "But they might provide a good starting place for editing to make the test more flexible." I'm well aware of the limitations of record/playback but this was more of a thought experiment of what else we could do if we were already implementing something (the detailed logging) that gave us most of the data we needed for a record/playback system.
    – c32hedge
    Commented Nov 2, 2017 at 19:43
  • 1
    I added a blurb about our experience with using record/replay vs PageObject tests.
    – Peter M. - stands for Monica
    Commented Nov 2, 2017 at 20:23
  • Thanks! The last paragraph in the update is particularly helpful.
    – c32hedge
    Commented Nov 2, 2017 at 20:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.