2

We have a group of "testers" (basically students etc., no professional testers and purely black box) who basically do not find bugs but rather use the app as expected in the live environment, something like end-to-end tests with different goals (perform 200 purchases or as much as possible and look for crashes etc.).

I would say the "ALPHA testing" is the closest one as this is a simulation of real use done by other people than developers or professional testers but still done in-house.

  • Seems to me like they are doing UAT... user acceptance testing. Hence - user acceptance testers? – KisnardOnline May 21 '15 at 21:12
  • @KisnardOnline Its certainly not UAT, since its not users who accept the application, its just random people. These users have very minimal say in the application functionality if at all. Sounds more like a closed alpha/beta test depending on the phase of the application. – Niels van Reijmersdal May 22 '15 at 14:06
2

There are different ways to classify testing.

If you want to classify it based on the level of testing, I would call it "system testing" or "end-to-end testing", since you are working with a system that is fully integrated already. Since they are working in an environment that is close to the live environment, they are not only making sure that the software is functionally correct, but also that it can function in the desired environment, which is something that lower levels of testing may not fully be able to accomplish.

If you want to classify it based on the type of testing, it could be "compatibility testing", since you are running it in an environment that mimics the live environment and you can ensure that users will have a positive experience. It could also be considered "acceptance testing" (probably "factory acceptance testing", since your employees are performing it instead of the customer or users), but this usually implies that it's specifically performed against the requirements to ensure the complete system works as expected and not more exploratory testing or ad-hoc testing. You may also be able to call it Alpha testing, since the use of an independent test team is acceptable.

Based on your comment, I would call the type of testing that is being performed "system ad-hoc alpha testing" or "end-to-end ad-hoc alpha testing". By including the term "system" or "end-to-end", you make it clear that the software and required hardware have already been integrated and likely been tested at a unit and integration level. The use of "ad-hoc" states that there aren't formal test plans or procedures that are being executed, but the testers may be given high level goals to accomplish through any means they see fit and report failures or issues. The use of "alpha" makes it clear that it's internal testing before the software is exposed to outside customers or users. Finally, it's important to include "testing" since you are still performing this work to find issues with the software.

Ultimately, defining the objectives of the testing and managing the testing process is more important than finding the right name for it.

  • As I mentioned, these are not real testers like the ones who perform system tests against functional specifications. These act as end users who just use the product with real life scenarios. Ad-hoc testing is still testing conducted with the goal of finding defects, our "testers" rather use the system as in the real world and report if there are crashes or bugs but they do not look for them specifically. As "Alpha testing" is done in controlled environment by indenepdent testers to simulare real world scenarios, it seems like the closest one..but FAT seems correct as well. – John V May 21 '15 at 13:14
  • @user970696 I don't know what you mean by "not real testers". It sounds like they use your system and look for crashes or bugs. That's testing and looking for defects. Given this information, I'm going to revise my fourth paragraph. – Thomas Owens May 21 '15 at 13:14
  • We have testers (performing boundary analysis, exploratory testing, etc..) and these "testers" who just use the product with no knowledge, not even the functional spec. They just use the product (it is a piece of HW with our SW) – John V May 21 '15 at 13:16
  • @user970696 So what? They are still testing your software before it goes live or is shipped. They get tasked with making sure your software doesn't have issues. – Thomas Owens May 21 '15 at 13:17
  • Sure, but in ad-hoc testing, you still aim at finding bugs by verification of requirements. In the alpha testing, it is rather real life scenarios you use to validate the product...I like FAT which you mentioned, makes sense! – John V May 21 '15 at 13:18
3

Be careful what you measure, because that is what are you going to get.

If you ask for 200 purchases, you will not get experience (and testing scope) what you would get by 200 different users making a purchase. You will get one user repeated 200 times. Which is better than nothing but does not tell you how your system will respond to 199 other ways to perform the purchase transaction. So it is almost useless.

There is a term "sapient testing". When sapient human is using website and looking for edge cases and bugs. It is very different from unleashing 200 monkeys bang the keyboards and report results which they find reportable.

Years ago our customer used '200 monkeys' approach to test the app we were developing for them (complicated application to handle non-performing mortgages - when people start paying, get behind on payments, get forbearance and/or refinance, add fees to principal, and possibly get foreclosure). They did not want to invest time in training students to understand what is going on. We got daily dozens of reports of "strange (for testers) behavior", of which 99.5% was correct behavior, which testers were not aware of.

All that was huge waste of time for developers to try understand what testers were doing (because they were also not trained to write good bugs, to save money on training as I said), so we get actionable bug report only after multiple emails exchanges. Then, we reported back that this is required functionality. After few weeks, futility of the approach became apparent, and 200 monkeys were transferred from testing to something more productive. Most of the results of their "testing" was discarded and not fully researched, because it became obvious it provides NO information.

Good luck!

2

The activity you are describing what I call "User testing" (assuming that the students are relevant representatives of actual users of the app).

Sound to me that the "User testing" is done in a testing phase that I would call "Alpha testing" which would have a milestone at the end, and a fix period, before "Beta testing" starts.

Beta testing would be testing with real users, real load etc. in a real-world app ecosystem. (Which could include testing of the payment model, support system, SLAs, error handling, help line, user manuals as well as the app itself).

2

Yes. That situation is defined as Alpha Testing. According to the ISTQB, Alpha Testing is:

Alpha Testing: Simulated or actual operational testing by potential users/customers or an independent test team at the developers’ site, but outside the development organization. Alpha testing is often employed for off-the-shelf software as a form of internal acceptance testing.

1

Typically you bring in real users for a usability test, but if they're looking for bugs, I'd say it's an ad-hoc manual test.

1

Lots of terms could describe this, depending on what you want to describe. For example you could describe:

  • the testers,
  • the potential problems they face,
  • the coverage,
  • the oracle(s) they used
  • the activities they perform
  • and/or their testing approach

Based on your question it seems like you are trying to describe the testers (or who's doing the testing), correct?

  • If the students are part of your target market you could say targeted user testing or just user testing
  • If they are subject matter experts than say that.
  • Alpha or Beta testing usually refers to the product's status or readiness but could also describe who does the testing.
  • Dog fooding, pair-testing, localization testing all sort of fit into describing the testers.

"Real life testing" could be called, as uTest refers to it as, In The Wild testing. Exploratory testing or ad-hoc testing generally refers to the approach (see the last bullet point in the first list).

Of course many of these terms are complimentary so you could say your company did "in the wild testing with a targeted group of users who performed user acceptance testing through an exploratory approach." I don't know why you'd do that but you could. =)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.