I am working on distributed JMeter testing.
When I run my tests during working hours it fails for 50 users with ramp up of 5 seconds.
When I ran the same test when everyone left the office after working hours, it worked perfectly.
My question here is:
Like you said in the comment above,
You have 50 users (employees) on the same network as you and are accessing the same server as the one hosting your application.
Out of these 50 at least 10 are trying to access the same application.
So, here is what I think you should do,
During your work hours when all your staff is working run the same script with thread count of 50 users. While the test is running in JMeter, try to open the same link in a browser on your system. Does the website return any server error? If yes, the server is not able to handle the load and chokes.
Now when your test completes execution, again try to access the website in your browser. If it opens, then definitely the problem is server not able to handle the load.
Now about after work hours,
Once all the employees have left the office your test for 50 users works fine. Now increase the load gradually. Try 100 users. If it works fine try 150 or 200 users and so on. If the tests start to generate error. Adjust the load by decreasing the number of users and run again. Adjust the load till you find the point where the your tests start to fail. If the numbers match with that when you run your tests during work hours, then you know that the server chokes at a certain load.
If the numbers don't match, then you should be checking to see the Network I/O and see if your network chokes when there are a certain number of system generating load. This will tell you whether the outbound requests are failing on your network.
TL;DR: Add better monitoring
Load testing goes hand in hand with monitoring. Load testing is a test to find bottlenecks and the limits of the application and its infrastructure. To be able to understand which part fails you need to have adequate monitoring on all levels.
In many respects, unless monitoring is carried out, the Load Test or Performance Test is likely to produce more questions than answers.
Each device and application in your chain should have monitoring:
Minimally you will want to have performance, access and error logs, so you can track the trail of the actions you are trying to load test and see which part is failing.
Load tests fail for numerous different reasons, the most obvious reasons are because of network/database connections limits.
Preferably do load-testing in combination with your infrastructure operators or system administrator and a developer to help you get the monitoring you need to analyze the bottlenecks and or fails. These people should have a good understanding how to analyze errors on servers and network devices.
Monitoring tools
Possible have a look at monitoring tools like NewRelic or Ruxit, these tools inject themselves on most levels of the chain. They visualize the data from the browser JavaScript calls till the database and back again.
Clean test environment
Why you are doing load tests during office hours? If you do not have a test environment to execute these tests you do want to make sure nothing else is making it harder to analyze your data. Any unknown traffic or actions will make it harder to analyze. It could be backups, torrent down-loaders or heavy manual database queries screwing with your excepted results.
Wonder why you are doing the load tests and create a setup that focuses on that. The most important reasons is to find your limits to know if your application will scale if the users count goes up. Finding out when you need to take action and scale the infrastructure.
Reads
Add a listener such as 'View Results Tree' and configure it to only log errors. You can then see what the exact response was.
Check the logs for the application under test to see if it threw any exceptions around the timeframe you saw the failures.
Ramping up to 50 users in 5 seconds does not give you much information beyond "it does not work". So slow down the ramp up rate.
The answers by Neils and TESTasy contain good advice about doing more monitoring. If you slow the ramp up to add one user every 30 seconds it will take 25 minutes to reach 50 users. During that time you will be able to see how the system is coping at lower numbers of users. The extra monitoring you are doing will also allow you to see which parts of the entire system (including the six items listed by Neils) are approaching overload as the load increases. It will also show how things change as the system reaches and passes overload.
TESTasy suggests using 100 and larger numbers of users out of hours. Great idea, but slow down the ramp up rate. Perhaps adding 5 users every minute. Then you can see at what levels the weakest parts of the entire system start to struggle.
I would suggest the following,
I also noticed that you are running a distributed instance of jmeter and you had some concerns in your question
network configuration should be same for the server where application resides and my server where i am running distributed jmeter? No it does not have to be on the same network configuration
i have 1 master and 2 slaves,how to understand which thread running on which slave? Please share a snippet from your result file so I can help you trace
