Alpha Testing via Company Testing Day?

Question

In a server-client environment we want to test / observe interaction of several clients and stress of the server. As it's a product the whole company is familiar with (we use it ourselves), I've thought, that apart from creating test cases with about one server and two clients run by dedicated testers, we could ask the staff to help us out.

Why not dedicate one hour for (nearly) everybody in the company to connect to the server at the same time and do some relevant actions, thus testing the architecture at a larger scale (more similar to our customers')?

1. Have you done anything similar?

If so:

2. Was it worth it?

If so:

3. How much time was spent? What percentage of your staff / which departments were involved?

4. What problems did you run into? (Especially were there problems you'd say are typical for this situation?)

EDIT:

Our testing manager actually considers a bug bash now.

Answer 1

This is a pretty common practice. At Microsoft we called it a bug bash I have also heard it called a bug hunt, I'm not sure if the industry has a standard term for it.

1. Yes, I and many other teams have done something similar.
2. Yes, we nearly always got some good data out of it.
3. As many people as we could get, usually for a few hours (1-4). We normally got pm, dev, writers, test and sometimes support staff to participate.
4. You need to be certain to outline the current list of known issues so you don't get flooded with a bunch of already known problems. You also want to give some examples of things to try so that people branch out from just the basic happy path and really explore. You also want to try to outline the new features and changes to previous versions so that you get extra attention to those things. Lastly, if you make it semi-competitive then it can be pretty fun and get more involvement. You can give awards for things like most bugs, most duplicate bugs (intentionally funny since this is not a good thing), most severe bug, most nit-picky bug, etc. Remember to reward everyone however, maybe with lunch or pizza or something to avoid problems seen here: Bug hunts and possible alternatives?

Answer 2

1. Yes, we've done something like this in the past.
2. Yes. For many reasons - performance issues, functional issues, team spirit, etc.
3. It's wasn't very time-consuming for us. QA, Dev, Customer Support were involved
4. Didn't really run into any problems

In addition to inviting others to participate, we used a load-testing tool to apply background load which ramped up in steps over time. For the first block of time no load, then moderate load, then large load, etc.

Folks doing the manual work were asked to note the specific time they found any issues so that we could correlate it with the load being applied at the time.

And we immediately followed it up with a discussion of what the functional aspects did as the system slowed down a bit.

All in all it worked well for us.

also see: http://www.allthingsquality.com/2010/04/automation-assisted-test-fest.html

Answer 3

1. Yes. This was part of process for major releases at a former company and was done in an ad-hoc way in other companies I was part of.

2. Yes. Apart from the reasons Joe states above, having people with different knowledge of the software combined with a different business view often resulted in issues/scenarios the development team hadn't thought of.

3. The amount of time varied. However, your idea of 1 hour is about perfect. Most people can put aside an hour to run around the software. Any longer, and you begin to be put people off. As for who and what percentage of staff, that is dependent on resource. If you intend to have everyone use the software in a meeting room, then number of seats is your limit. If they are to use the software from their desk, then every staff member could potentially be invited

4. There are two main issues, one of coverage, and the second of reporting (be it an issue, or conclusions). For coverage, you can split the software to business areas and point people at each one (in order of business importance). For reporting, you will receive a plethora of different quality of reports unless you set up a formal report procedure. Usually with this, I send out an email with the major heading for people to use in their report. That way they can just hit reply.