What kind of usability testing can I perform when the final users of the product are developers?

Question

I have built a framework aimed at developers and I'd like to carry out a test with users. However, since my final users are developers, I don't know how to approach this testing. The usual list of tasks to perform doesn't seem appropriate in this context.

Some context about the tool:

There are several APIs out there having a similar functionality but being different enough so it's not trivial to use them together. I've developed a framework that allows the user to forget about this integration, focusing on the results returned by the APIs combined. This way, developers don't need to spend too much time integrating the different APIs into their system, they just need to implement a common interface (if it's not implemented yet), launch my framework in a server of their choice and send their requests to it.

Considering this, is there any kind of test which fits best for this situation?

In the end, I want to test if using my system is a better/easier alternative to integrate the APIs by hand. I don't know if I should make a list of tasks to accomplish using the tool and then ask some questions, or if I should make someone code something using the API and without using it and compare the resulting codes (checking static code metrics, for instance), etc.

Any help is appreciated.

Answer 1

Interesting task. You could run two usability tests with developers:

• one with developers using APIs
• one with developers using your framework

and collect statistics (like satisfaction on Likert scale) and feedback on challenges (from think-aloud protocols) from both of them and compare. You care about both quantitative data (how good your framework was) as well as qualitative results (at what parts it was better and where it still needs some improvement and why). I once collected a lot of useful feedback about Web service search engine by just sitting together with 16 developers (in a separate 16 sessions) as they were using the assigned search engine to find a Web service and listening what they want to do and what they struggle with (this is form of thinking-aloud protocol).

There are many parts that contribute to how usable an API or framework is: design/architecture, examples, documentation, IDE support. Microsoft has performed usability tests for one of their libraries.

I have also ask similar question about this topic here.

Answer 2

Any kind of Beta testing is what you are looking for. How you approach that is totally up to you. Some of the options:

1. Organize a Hackathon ask people to use the framework,
2. Maybe you can find a local community of programmers to which you can present the framework and ask them to use it (Where I come from there are some local Java user groups that meet on occasions),
3. If you do not mind the bigger public - find some internet community that brings together people that tend to use those APIs (internet forum, reddit, maybe somewhere on Stack),
4. Use it yourself - I know it is not the 'best' solution, but 'giving it a go' might help you understand its limitations (this, I guess, classifies as Alpha testing)

Then collect the feedback, simply ask people what they liked about the framework, what is missing from it, what they would improve. Once your Beta group is big enough you can do some static code anlysis and see which methods were prefered over the others.

Answer 3

From your question, I gather that you are making a sub-product that can be integrated into a larger product and creates an adaptation layer to unify several other APIs. The usability metric that you want to test is the ease of creating a system that includes your product compared to creating a system that doesn't.

Usability in this sense is not what is normally meant when people talk about usability testing. What you really are asking is how well does your product meet its purpose and that can only be really tested by putting it out in the wild and seeing how well it gets adopted by your target audience.

You can get early feedback by using it yourself, for example to create a demo system to show others what the possibilities are, and by asking experienced developers in the field you are targeting to try your product and asking them what needs to be changed/improved before they would recommend it for their next project.