Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

We have a Websocket server which I have to load test. The test case is as follows:

  1. Maximum number of connections possible
  2. Each connections should be sending 1 message/second

Tools I have tried:

  1. Phantom JS (Unsuccessful)
  2. Apache JMeter (Partially successfully): I was able to simulate simultaneous connections. For each connection a separate thread is created. But I am only able to send one message after connecting using the plugin https://github.com/kawasima/jmeter-websocket/ the thread just remains stagnant after send that one message. I cannot create a separate thread group for sending messages as I wish to test a situation where communication takes place via the existing Websocket only. I might be missing some configuration or a rare plugin which allows me to recreate my test case.
  3. NeoLoad 4.2: This is the exact tool I was looking for. It allows me to perfectly create my test case BUT my company is not ready to invest in any testing tool(Forget about having a testing team either). And due to trial limitations, I am only able to simulate 10 users.

Please suggest/advice with any possible tool or plugin which will help me.

EDIT:

Here is how I managed it.

public class Tester {

    static int i;
    int numberOfConnections = 300; //number of simultaneous connections.

    public static void main(String[] args) {
        final String channel = "111";

        for (i = 0; i < numberOfConnections; i++) {
            try {
                TimeUnit.SECONDS.sleep(3);

                new Thread(new Runnable() {

                    @Override
                    public void run() {
                        try {
                            final CUstomWebsocketClientWrapper client = new CUstomWebsocketClientWrapper("http://162.243.235.172:8080", "demo", "demo");
                            client.createConnection(channel + i); //Open the connection

                            TimeUnit.SECONDS.sleep(2);

                            new Thread(new Runnable() {

                                @Override
                                public void run() {
                                    while (true) {
                                        try {
                                            client.push(channel + i, "Message" + i); //Send message
                                            TimeUnit.SECONDS.sleep(1); //Change message sending rate here.
                                        } catch (InterruptedException ex) {
                                            Logger.getLogger(Tester.class.getName()).log(Level.SEVERE, null, ex);
                                        }
                                    }
                                }
                            }).start();

                        } catch (InterruptedException ex) {
                            Logger.getLogger(Tester.class.getName()).log(Level.SEVERE, null, ex);
                        }
                    }
                }).start();
            } catch (InterruptedException ex) {
                Logger.getLogger(Tester.class.getName()).log(Level.SEVERE, null, ex);
            }
        }
    }
}

This will create clients sending messages at the defined rate. Make sure you have loads of RAM or you'll end up with an out of memory error. And after each test you will have to restart Tomcat/Jetty so the threads are destroyed. This is not the ideal or memory efficient way but this is the only method that works for free.

share|improve this question
add comment

4 Answers 4

up vote 1 down vote accepted

In the past, I used a .net websocket client library, however .net 4.5 has support for websockets: http://www.codeproject.com/Articles/617611/Using-WebSocket-in-NET-4-5-Part-1

I am sure there are other libraries in java or other languages that also allow you to interact with a WebSocket, but I am less familiar with them. Once you find a library, you can create the calls to make the connection and then send any additional requests to the WebSocket.

You could then either use a tool such as Jmeter through it's support of executing junit tests, or write some code to spawn threads and send requests yourself.

What I ended up writing through the use of the .net WebSocket library was a functional automation framework for the WebSocket that was extended to also perform load and stress tests. All of the tests were executed through nunit, and our framework handled all of the thread management. I'm not sure what your or your team's skill set is, but if you have a developer familiar with writing a multi-threaded app, it isn't so bad to write it from scratch. The nice thing about this approach is that you can easily use it for functional automated tests.

share|improve this answer
    
This is actually what I ended up doing. I am the developer and tester both. There are Websocket libraries in Java and I am using this github.com/Atmosphere/wasync But the program fails to create a consistent test scenario. You are right, I need to look more into multi-threading properly. –  Gaurav Bhor Jan 22 at 6:29
add comment

What can I see from your code is that at least you generate 600 native JVM thread to simulate 300 connections only which is a small number, I also doubt that the library you are using CUstomWebsocketClientWrapper creates extra threads per each connection which will end up with super inefficient tool to do the job.

I tried the recipe here http://blazemeter.com/blog/websocket-testing-apache-jmeter which is different from your experience since it depends on a different Websocket JMeter Sampler plugin. The problem as I mentioned earlier was that the plugin internally creates ~10 threads per user which is inefficient for me to simulate 60K from a single box

What I'm trying right now is using the thor https://www.npmjs.org/package/thor, it seems very light weight I used it to simulate 10K concurrent connection so far.

share|improve this answer
    
Do you just want concurrent users or concurrent active users? I know it is not memory efficient. But creating 2 threads instead of 10 is better. Also, I had to simulate using different terminals. Creating 60k threads from one machine will freeze your jvm. Please try distribution.. –  Gaurav Bhor Jul 3 at 10:13
    
Of course I will use distribution to simulate 100K concurrent users, but with your algorithm, how many machines you need to achieve this number; you need 200 machine to simulate 60K!, Did I get it right? –  Montaro Jul 3 at 12:07
    
I could simulate 700-800 connections from one terminal. 4 gigs of RAM each. Also, how much RAM does your server have? If you are considering paid solutions, Neo Load is the best for this task. –  Gaurav Bhor Jul 3 at 12:12
1  
I executed this on my machine: arefaey@arefaey-MacBook-Pro ~> thor -C 60000 --amount 60000 ws://localhost:9000 -P 13 --masked -W 4 This spawns 4 node workers and the maximum memory footprint for all node processes was almost 1GB –  Montaro Jul 3 at 12:49
add comment

There is a newer plugin for JMeter which is fully RFC 6455 compliant and allows re-using same TCP session for multiple duplex requests.

share|improve this answer
    
That's an amazing step by step guide! I'll keep it marked for the future. For now, I have written a multi-threaded program which servers just fine. :) –  Gaurav Bhor Mar 8 at 8:03
    
@GauravBhor Can you please share your code if it is just a test suite? –  Montaro Jul 1 at 15:02
    
@Dmitri T, I tried this tutorial, my problem as I commented there that each Thread/User is actually generating 9-10 native JVM threads, do you have any advices? –  Montaro Jul 1 at 15:03
    
I guess it worth asking plugin developer, perhaps underlying Jetty implementation may be the cause as looking into the plugin code I don't see anything spawning 9-10 threads. –  Dmitri T Jul 1 at 15:29
1  
@Montaro I've edited my question to include the code for you. Let me know if you need anything else. –  Gaurav Bhor Jul 2 at 6:09
add comment

You may use websocket client library for any popular language (e.g. java or python). For your test scenario it will be more suitable way of testing, rather than try to find some test tools.

Also, I'd advise to use 'select' or 'poll' instead of threading, because it scales better, especially if you need more than 10K connections. It took about a couple of hours for me to create simple test suite using python asyncore library.

share|improve this answer
    
Alright. I'm not sure in which context are you saying 'select' or 'poll'. Could you please explain? –  Gaurav Bhor Jan 24 at 14:20
    
I was talking about I/O multiplexing. The couple of low level functions are 'select' and 'poll'. –  algo_dogs Jan 28 at 17:22
    
In Java, that'd refer to the NIO library with socket channels –  ThorinII Jul 4 at 11:55
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.