Many teams do not deliver GUI but a library or a service that is used by other developers. For instance:
A company delivers a library to be used by other developers and testers: it can be as big as JDK (Java Development Kit) from Sun/Oracle or Selenium Web Driver.
Your team provides a microservice to be used by other teams in the company.
All those artifacts can have user experience (UX) bugs similarly to GUI -- the name of a method might be misleading and do something else than the name suggests, or finding a class to do X might be hard because of insufficient documentation or wrong abstraction level.
I sometimes complained about such bugs to developers but I have never performed regular usability tests for APIs. I've found that Microsoft did. The approach they took was to involve real user in task-driven tests, just like you do in normal (GUI-based) usability tests, with one small difference -- their end users were developers. You give a development task to a developer to be performed with your software and you observe how they handle it.
I wonder whether other testers do perform usability tests of API?
In the end you would need to get into shoes of a developer or involve them in your tests.