We are starting from scratch on how to get better visibility of how and what our software suppliers are actually testing and the governance around that process. Would welcome any input or examples of what has worked. No point in reinventing the wheel
2
-
Are you trying to solve a concrete problem or prevent something that has happened before? So what is the need and why? Or just wanting to check-up on your supplier?– Ray OeiCommented Nov 23, 2017 at 21:40
-
We have had cycle one delivered from suppliers and return rate of defects has been high. So are looking into creating governance for cycle 2 delivery, so that we have great insight in to their off site testing and a firmer 'hand over' to us. The ultimate aim is to reduce the effort on testing with us and a reduction of returned defects– Mike croftCommented Nov 24, 2017 at 8:34
Add a comment
|
5 Answers
You can request the supplier to share their test documents like Test scenarios, Test Cases, use Cases which they used while testing. By reviewing these documents, you can understand how much the testers have tested or going to test.
You may also request automation scripts if you have any technical person to review them. You may ask for results of the tests conducted.
Talking to the Tester regularly will help you to understand how much testing is done and on what grounds. Also, your feedback to the tester about the software application will surely help the tester to explore more test scenarios.
This is a common practice. Many customers audit their suppliers processes and quality standards. Though this is usually done whilst selecting suppliers.
I've been on both sides of the fence where the team I am working have audited a supplier and the company I've worked for has been audited.
When this has happened to me, I've been expected to demonstrate that I adhere to my company's quality procedures. I maintain my tests in a controlled fashion and the tests offer value.
It's not unreasonable to ask what tests your supplier runs. You expect to see developers continually running unit tests. You would also expect that you supplier has tested before releasing to you.
In terms of reducing testing effort on your part, my opinion is that you should still test the software and the users should run UAT.
Your supplier will be focused on system tests, whereas your organisation ought to test and the users are concerned with making sure the software meets their business needs.
For example, one of the projects I am working now, the vendor tests the software and share the tests. I have written my own functional tests whereby I check each option works. By this point, the expectation is that during UAT, our users will not suffer from frustrating crashes, broken links etc. This leaves them to focus on solely making sure the software meets their needs.
Where software is bug-ridden and/or crashes a lot, you find they very quickly lose interest and can disengage. This happens as users have to stop doing their 'day job' to run UAT and when they make time, they get frustrated when things don't work.
-
Thanks Chris. Yes I can see that we will need to have greater visibility as to what their Developers are testing and following on, what system testing they are performing. Do 'they' supply you with their test plans, test scripts etc in advance ? Guess the position I want to get us in to, is that when we get a defect during UAT we can trace back to see if that scenario was covered off by the supplier. If it did and did not fail - what is different now. If they didn't - then why not ? Thus reducing the age old change request v Defect debate Commented Nov 24, 2017 at 11:48
-
Yes, I have seen vendor test plans and test scripts. It helps us (as customer) understand what has been tested and where we need to focus. You're right, the traceability aspect is really important. Our testing tends to have more breadth as we are typically dropping a new system into a business area, so we have to consider system integration as well as the NFRs such as performance, stability and resilience. Commented Nov 24, 2017 at 13:31
-
1The problem I have with focusing on standards and processes is that your real aim: getting software that behaves the way you want, often tends to get lost. Don't forget that if you put more 'paper trail requests' on your supplier you are going to pay for it in some way. So I would focus much more on what went wrong. Was it obvious wrong? Was it misunderstanding of what was agreed upon? Misaligned assumptions and expectations? Did you describe good enough what was needed? Were you able to check whilst it was developed? Do they feel things have gone wrong? Or are they just plain bad at it?– Ray OeiCommented Nov 24, 2017 at 20:54
Just to throw in my two cents and add to the existing answers - you can (and definitely should) ask for proof of testing, but they might not be contractually obligated to share that information.
Alternatively, you might get lucky and have some really open and honest suppliers who don't mind airing their dirty laundry.
I've seen it both ways, when asking for the exact same thing. We suggested they host daily stand-ups, share defect logs, generate spreadsheets, etc. and one company didn't share anything, whilst the other shared absolutely everything.
-
1"one company didn't share anything, whilst the other shared absolutely everything", yep. Seen that too. Where companies won't or don't share it makes the customer/supplier relationship difficult and trust issues come into play. Quite often suppliers fear being beaten with a big stick, so are not open. But, when open, both sides can take actions and can lead to a productive relationship. Commented May 30, 2018 at 10:33
I like checklists. Because you can see all actions that testers do from their side of work. And as a developer, checklists can give you an extra information. Like, if you have columns with modules or integration - an aggregation of major code parts. So developer can see as a white box inspector which buggy module can influence other modules.
Any company I worked, they used some kind of variations of this checklist template. I add some of my new ideas (with module and integration). Quality control and inspection plan
The other way of ensuring it to ask for module wise test results on daily basis, be it an automation test reports in simple email format or a spreadsheet with daily test round report, you can have an independent testing team/organization that can help you understand , what has been done in a testing point of view for testing of product, or if the company is open, you could ask them to share document covering test scenarios, test case, and RTM, but for this some companies might bill you extra charges.
answered May 30, 2018 at 13:38
