Can some one give me proper idea on how to perform a Root cause analysis of a performance issue.
Eg: examine hardware bottle necks,Connectivity issues,SQL server performance,examine the code via white box text ...etc.
Given you are talking about everything from Hardware to SQL Server to Code, I'm assuming your performance issue is being observed at a high level. i.e. you have no idea what component(s) may be at fault.
Here are some ideas of approach when you're not sure where to look:
- Recreate and Isolate - By this, I mean try to create the simplest reproduce case that exhibits the issue. Removing and Adding variables to identify the conditions that create the problem. Sometimes the reproduce workload will point you in right direction (i.e. only happens under load, or when large data volumes are used, etc.) Simple(minimal variables) and repeatable are goals here.
- Monitor - Correlate system health with symptoms being seen. Introduce tracing where you can to give feedback. Looking for correlations here. i.e When things perform bad the cpu is pegged on this server, or when things perform bad we are running a particular process, etc.
- Educated Guess - You or others may have a hunch as to where things are breaking down, hone in on that first. Protect against assumptions, but have to start somewhere.
- Persist/document - My experience is that when first setting out on tackling a performance issue, early in the process your efforts seem to raise more questions than answers. You start to question your own sanity. Document along the way, keep records of what you have tried and what you saw. Another tip: tell your story to those that will listen, by explaining to others what you have tried in your attempts to find root cause, it can add clarity to yourself when you feel like you're going in circles.
If you know what area is at fault (DB vs Servers vs Software vs Network. etc.) then each one of these have their own technical approaches to narrowing down. But, I stress again, isolate to simplest recreate. Having a repeatable recreate in hand, imo, makes the analysis possible.
You covered some possibilities, plus db queries can also be an area of focus. I'd recommend starting at the lowest level (code) and working up, so as to eliminate as many variables as possible at each step along the way. A code profiling tool will be your friend here, like ReSharper dotTrace (C# world), and it should help you identify hotspots rather quickly.
Normally the process of identifying the bottleneck includes the following checks:
- Baseline OS health metrics (CPU, RAM, Network, Disk, etc.). Your application can respond slowly due to simple lack of resources and upgrading the hardware will eliminate the issue. See How to Monitor Your Server Health & Performance During a JMeter Load Test article for more details.
- Infrastructure configuration. It might be the case that your middleware hardware or software isn't suitable for high loads. The absolute majority of web, application, database servers, load-balancers, etc. default configuration is fine for development and/or debugging but you need to tune it additionally to get the maximum performance.
- The issue could be in your application code. It is recommended to run performance test under profiler tool telemetry - it will allow you to detect slow functions, largest objects, memory leaks and so on.
- Carefully inspect all logs and traces you can find, the bottleneck can be caused by i.e. wrong DNS resolution, faulty network router, broken cable, delay of CDN response, whatever.
srini,Root cause analysts is mechanism ,to find the cause ,we can go throw dig the defect ,the defect either it will be requirement missing ,code level missing ,testing point of missing and also product defects also missing these are the ways missing defects are root cause analysis .
By using root cause analysis where we are miss the defect why we are missing .we need the given the clarification to the end users .