Best practices for automating manual test cases in TestRail

Question

Note, this is specific to TestRail but some questions could be considered generic issues:

I have worked for a couple companies that have gone through the implementation of automation into TestRail. In each company, there has always been different practices of doing things.

• When you automate a manual test case, what do you do with the manual case and what do you do with the automation case?

  • example 1: Do you keep the manual test and just flag it as automated, if so do you point to the case ID in automation to have a link and add any other fields to the TestRail case? How do you keep it up to date if the test changes?
  • example 2: Do you delete the manual test and import the automated test in its place?
  • example 3: Do you keep both tests but flag the manual one as automated?

• What do you use the reference field for?

  • example 1: Do you use it to link the JIRA epic/story for the test case? how do you then link to the requirements?
  • example 2: Do you use it to link to the requirements, if so, what tool do you use?

• Most automated tests are WebDriver, API, etc., but do you include unit tests in TestRail?

• If I have a manual test that's been automated but not under each platform, what do you do?

  • example 1: Don't flag it as automated but make a note in the test stating it is automated for xyz platforms (or a separate test case field with automated platforms).
  • example 2: Flag it as automated and spin up a new test case for the non-automated platforms.
  • example 3: Not a concern, if it works on one platform, it should be sufficient.

• If you add automated tests to testrail that do not have a corresponding test case, where do you put them?

  • example 1: Use a section ID of the feature to place them where they should logically go.
  • example 2: Use a section ID of the feature but then have an 'automated' folder where they go.
  • example 3: Have a flag automation folder.

• How do you work around being reliant on testrail IDs?

  • example: if you migrate from a cloud solution to an on-premise solution, all of your IDs break. If each test case is linked to an ID, this would be thousands of code changes needed; or if a manual tester moves sections around to reorganize the group tree, it may break automation.

Answer 1

I can't speak directly to TestRail but can to the conceptual questions asked.

When you automate a manual test case, what do you do with the manual case and what do you do with the automation case?

The manual test case should be stored in a SQL like or NO SQL like database. One time one place so everyone can edit, update, insert or delete tests. The automation work (code) should be associated with the manual test case so that a query like "Get me all Automated tests for XYZ" return all the Manual Tests. By reading the manual tests you will know what the automation is supposed to do.

Most automated tests are webdriver, api, etc.. but do you include unit tests

Unit Test are automated tests, but I think what you mean is Automated GUI tests and Unit Tests, do they both go in same places? The answer is usually not, True unit tests should be in same code repository as source code for application, one folder away. GUI tests are not usually included in developers code mostly because they are only related in what they test and are usually developed completely separately.

If I have a manual test that's been automated but not under each platform, what do you do?

There should be meta data associated with the automation such as something like "Category" or "Release" of "Platform". This metadata must be queryable so that you can do this "Get me all Automated Android 10.0 Tests for component XYZ". The effort of getting the other releases is just a regular ALM / User Story concern.

If you add automated tests to testrail that do not have a corresponding test case, where do you put them?

Id depends, true unit tests belong in same place as application code, GUI Automation belongs anywhere you decide is best. If there is no corresponding Manual test, then you should be able to query these just as easy as any other test (Automated or not).

How do you work around being reliant on testrail ids?

All repositories should be in some sort of database. But you may not have access or control over the IDs. In that case just don't try to alter them as they are acting as unique keys. Let the id's become meaningless to you and rely on naming conventions over configuration instead. Naming convention where you do have control.