What type of testing should I be starting with for a big PHP site?

Question

My title explained it all. I have a site to be tested which has almost 100 million lines of codes. This site hasn't been tested yet. I don't know what type of testing should I start with. Should I start with some automation or manual testing? I thought of testing with QTP, but before that thought of consulting with the greatest experienced tester community.

Answer 1

With something like this, that's huge and has no prior testing, I'd say that your first priority would be exploration (most likely manual) to both gather information and to determine the best approach to build regression for it.

I'd approach this way:

1. Manual exploration, focusing on independent clusters of functionality and testable material. Some of the things I'd look for include data validations (how they're handled, whether they need to be tested at multiple levels such as web-front, database, etc), interfaces between modules, common navigation and process flows, areas that fit the 80/20 rule (the 20% of the application that will be used by 80% of users).
2. Prioritize modules/function clusters/process flows (depending on the nature of your application, any of these could be the most sensible base for your regression test cases - I work a lot with point of sale software, so a process flow based around "sell something" is my most common test case) - you probably want to work with a subject matter expert to determine this. Also seek out areas with the biggest potential return on automation time. No matter what there will be a large up-front cost to automating, so choosing high-priority test cases that are a) mission critical, b) relatively simple to automate, and c) have a high regression risk is a way to strategize (an example from my experience - the software I test has to support tax rules in multiple different countries. The short-form tax regression automation is in two separate automation projects and runs for a total of 16 hours. The same regression would consume 5 testers for over a week and be much less accurate - but it took a LONG time to build and generating accurate baselines was pure misery. That said, it's saved my bacon often enough to be worth every minute of development).
   1. Choose a tool that makes sense with the application you're working with. There are numerous excellent tools for web application testing, but given the size of the application and the use of PHP, I'd guess there's other interfacing involved, such as databases. You want your tool to access that as well, both to store test data (reading a text file also works for this. I'm working with test harness in the form of a relational database built in CSV files, and a data store that's its own relational database in CSV files). If it can reliably do what you need, then any tool is good - but you need to make sure you (and anyone else involved) knows how to use it.
   2. Mix exploratory testing and building of automated tests within each module in turn, starting with the top priority and working down.

Most important, make sure that everyone you're working with knows this is not going to be fast, easy, or cheap (if only in terms of time invested). You need to make sure the people you're working with know what to expect in terms of increased complexity generating increased testing and automation time.

Good luck!

Answer 2

You (or better application owner) should split application into function zones/features by priority. Proper organize your tests which makes easier maintain them. Type of testing depend on application itself. Some application behavior is more efficient to test with automated tests (especially when it be planed to repeat test time to time). But there can be web application features which could be not efficient to test with automated tests (hard to write automated tests for it or it takes lots time). So in reality mixed testing styles often used. Which automated testing framework you choice depend of your personal experience and needs.

Answer 3

100 million (codes? lines? or something else?) and the site hasn't been tested at all?

You should consider some manual functional testing first. No sense in looking at automation if you aren't even sure the site works at all.

Do you know the requirements? Is it at least clear what the site should be doing?

Answer 4

Well, who are you testing it for?

What do they want to know, in general? What kind of decisions will they make as a result of the information you find for them?

What oracles do you have - i.e. what can you use to help you understand when you might be seeing a problem? Don't get too stuck on "requirements" - if it's never been tested ever ever, then I doubt you have any. Think laterally - you could make use of people like product owners, users or support staff who you can ask about desired behaviour, documentation such as help guides, comparison with competitor's sites (industry standards), and so on.

Do you have anyone else to help you with this testing? Are there any tools that you can use to help you, and what kind of testing environments do you have available, if any?

How soon does your client want to get this information? And in what form? And how often? If they're not too sure, you might want to suggest that you start out by giving them a sample report quite early, and then they can figure out what they like and don't like and let you know.

Testing is an investigation: if you don't know what your client wants, or worse, if your client doesn't know what they want - you have little hope of providing them with useful information. Figure out your mission first: I use my own mnemonic to help guide my questions, SORT (Scope, Oracles, Resources, Time) - you might notice that's how I've structured the questions above. If the client has never had any experience of testing before, you may need to guide them a little by suggesting options. Whatever you do, it's essential to ensure that you review with them on a regular basis to make sure they're still happy.

For any testing project, you should be starting off by discovering this kind of information. If you have that, let us know. If you don't have it - go find out! (You can begin testing once you know some of it - it's likely to be a gradual process and you'll probably uncover more and more as you go, but if you don't know ANY of the above, you should be talking to your client.)

Answer 5

The other posters have made excellent points. When you get to where you are ready to start adding automation to the web site testing, I have found WATIR to be extremely helpful:

http://watir.com/

Answer 6

My first question would have to be why did you let it get to 100 million lines of code without testing anything so far? Have any of your developers done any unit testing at all?

If your developers have carried out some unit testing then you may be out of the starting blocks already. Build this up with some exploratory testing initially. Whilst you're doing that look at creating a more structured test plan which will allow you to prioritize your testing.

I'd agree with the post above that you could start by breaking the app down by functional area. Then talk to the developers and end users. From the end users get a feel for which parts of the application are most critical to them and which areas will be used most frequently. From the developers find out which areas/features of the code were the most complex to implement. From here you can list the functional areas by priority and risk.

You've got a lot to do here. So if you can I'd engage the end users early in this too. That is assuming that they don't mind seeing the unfinished, untested product. If putting a poor quality product in front of the customer early on is likely to give you bad press then avoid this. However, if they don't mind helping out with a bit of feedback then it's another few pairs of hands to the pump.

Personally I'd steer well clear of automation until you've got a good manual test process in place first. If you don't understand what you're testing from a manual perspective automation will just add another layer of complexity. Wait till you've got a good feel for the application then you can start thinking about which areas of the application will be well served with automated testing.

Hope you've got plans to get a good team of testers on board to help you with this.