My title explained it all. I have a site to be tested which has almost 100 million lines of codes. This site hasn't been tested yet. I don't know what type of testing should I start with. Should I start with some automation or manual testing? I thought of testing with QTP, but before that thought of consulting with the greatest experienced tester community.
- Anybody can ask a question
- Anybody can answer
- The best answers are voted up and rise to the top
With something like this, that's huge and has no prior testing, I'd say that your first priority would be exploration (most likely manual) to both gather information and to determine the best approach to build regression for it.
I'd approach this way:
Most important, make sure that everyone you're working with knows this is not going to be fast, easy, or cheap (if only in terms of time invested). You need to make sure the people you're working with know what to expect in terms of increased complexity generating increased testing and automation time.
100 million (codes? lines? or something else?) and the site hasn't been tested at all?
You should consider some manual functional testing first. No sense in looking at automation if you aren't even sure the site works at all.
Do you know the requirements? Is it at least clear what the site should be doing?
Well, who are you testing it for?
What do they want to know, in general? What kind of decisions will they make as a result of the information you find for them?
What oracles do you have - i.e. what can you use to help you understand when you might be seeing a problem? Don't get too stuck on "requirements" - if it's never been tested ever ever, then I doubt you have any. Think laterally - you could make use of people like product owners, users or support staff who you can ask about desired behaviour, documentation such as help guides, comparison with competitor's sites (industry standards), and so on.
Do you have anyone else to help you with this testing? Are there any tools that you can use to help you, and what kind of testing environments do you have available, if any?
How soon does your client want to get this information? And in what form? And how often? If they're not too sure, you might want to suggest that you start out by giving them a sample report quite early, and then they can figure out what they like and don't like and let you know.
Testing is an investigation: if you don't know what your client wants, or worse, if your client doesn't know what they want - you have little hope of providing them with useful information. Figure out your mission first: I use my own mnemonic to help guide my questions, SORT (Scope, Oracles, Resources, Time) - you might notice that's how I've structured the questions above. If the client has never had any experience of testing before, you may need to guide them a little by suggesting options. Whatever you do, it's essential to ensure that you review with them on a regular basis to make sure they're still happy.
For any testing project, you should be starting off by discovering this kind of information. If you have that, let us know. If you don't have it - go find out! (You can begin testing once you know some of it - it's likely to be a gradual process and you'll probably uncover more and more as you go, but if you don't know ANY of the above, you should be talking to your client.)
You (or better application owner) should split application into function zones/features by priority. Proper organize your tests which makes easier maintain them. Type of testing depend on application itself. Some application behavior is more efficient to test with automated tests (especially when it be planed to repeat test time to time). But there can be web application features which could be not efficient to test with automated tests (hard to write automated tests for it or it takes lots time). So in reality mixed testing styles often used. Which automated testing framework you choice depend of your personal experience and needs.
The other posters have made excellent points. When you get to where you are ready to start adding automation to the web site testing, I have found WATIR to be extremely helpful:
My first question would have to be why did you let it get to 100 million lines of code without testing anything so far? Have any of your developers done any unit testing at all?
If your developers have carried out some unit testing then you may be out of the starting blocks already. Build this up with some exploratory testing initially. Whilst you're doing that look at creating a more structured test plan which will allow you to prioritize your testing.
I'd agree with the post above that you could start by breaking the app down by functional area. Then talk to the developers and end users. From the end users get a feel for which parts of the application are most critical to them and which areas will be used most frequently. From the developers find out which areas/features of the code were the most complex to implement. From here you can list the functional areas by priority and risk.
You've got a lot to do here. So if you can I'd engage the end users early in this too. That is assuming that they don't mind seeing the unfinished, untested product. If putting a poor quality product in front of the customer early on is likely to give you bad press then avoid this. However, if they don't mind helping out with a bit of feedback then it's another few pairs of hands to the pump.
Personally I'd steer well clear of automation until you've got a good manual test process in place first. If you don't understand what you're testing from a manual perspective automation will just add another layer of complexity. Wait till you've got a good feel for the application then you can start thinking about which areas of the application will be well served with automated testing.
Hope you've got plans to get a good team of testers on board to help you with this.