Reputation
3,556
Next privilege 4,000 Rep.
Access 'trusted user' tools
Badges
2 11 33
Impact
~176k people reached

Aug
19
reviewed Approve Monitoring DB setup in a way convenient for DBA
Aug
18
revised Monitoring DB setup in a way convenient for DBA
improved formatting, provided code snippets
Aug
18
asked Monitoring DB setup in a way convenient for DBA
Aug
18
accepted Checking two related functionalities in same test case
Aug
18
comment Is branching tests the only option for system with multiple versions?
Cost of maintaining branches will be higher, if test suites for each release/major version aren't stable (e.g., we are currently increasing coverage in end-to-end tests), so every change in one branch must be merged into another. I think a good moment to create branch would be when test suites are relatively stable for one release/major version, so only bug fixes are merged.
Aug
18
comment Is branching tests the only option for system with multiple versions?
In your 1st approach (1st paragraph) didn't you have test fixtures specific for a given version of a system? If so, then I guess those fixtures weren't tightly coupled to the system API, classes, DB so you had no compilation, runtime problems, correct?
Aug
18
comment Is branching tests the only option for system with multiple versions?
That's a good idea, but will require QAs and development work more closely to immediately resolve sync problems (e.g. compilation failures). We do so for unit and integration tests, but end-to-end tests we versioned in a separate repository. Anyway, might be worth considering merging both repositories together, thanks!
Aug
18
comment Strategies for automating testing on features that are only on certain environments
Technically, NUnit frameworks provide a way to group or select tests to run. E.g. In Java-world TestNG provides XML-defined test suites, JUnit provides TestSuite annotation to group tests and Category annotation to select them.
Aug
17
revised Checking two related functionalities in same test case
added 4 characters in body
Aug
17
comment Checking two related functionalities in same test case
I really like the 3rd idea. I think I might use hard assertions for more important functionality and soft assertions for less important one. A failing hard assertion will immediately stop test execution, while hard assertion success will let soft assertions to provide feedback on reporting functionality... This might be a way to implement your 3rd idea. I just nead to read more on TestNG/JUnit result output format.
Aug
17
revised Is branching tests the only option for system with multiple versions?
edited title
Aug
17
comment Checking two related functionalities in same test case
Joe, usually our test report is more torough (metrics are accompanied with defects list and their severity). However, this time I had less time to prepare such report (and create defects), so I started to think how test framework and test design can help provide clear feedback more quickly.
Aug
17
revised Checking two related functionalities in same test case
Added information about using reporting functionality for troubleshooting
Aug
17
comment Checking two related functionalities in same test case
Regarding 1st idea: Our managers are really reasonable people with much experience in both testing and development, so they ask questions about details, especially when the news are bad :-) It just takes me more time to answer those questions, so I was thinking how to improve logging those errors.
Aug
17
comment Checking two related functionalities in same test case
@user246, actually this is functionality (separating setup for tests in BeforeClass annotated methods from actual test cases) is common feature in many XUnit frameworks. But JacekM applied this very nicely!
Aug
17
comment Checking two related functionalities in same test case
Still, JacekM's idea made me thinking about something like for(Functionality1).verifyThat(...) were verifyThat is soft assertions and results for different soft assertions are logged by the test framework on separate stacks and then reported separately... Somehow... Need to think how to implement that on TestNG.
Aug
17
comment Checking two related functionalities in same test case
+1 That's a good idea to separate test run from checking test output with assertions. It will not work for some more complex test cases with multiple steps interwoven with assertions (step 1, assertion 1, step 2, assertion 2). Grouping all assertions together after step 2 would make no sense, because step 2 may override the system status after step 1 and assertion 1 will always fail.
Aug
16
revised Is branching tests the only option for system with multiple versions?
added 2 characters in body
Aug
16
asked Checking two related functionalities in same test case
Aug
16
asked Is branching tests the only option for system with multiple versions?