I would suggest possibly assigning some "free play" time with a minimum number of participants, specific classes of player or levels of experience and with a number of objectives and/or story points from the core functionality and from the latest couple of sprints to try to achieve within the time period.
This could be followed by a structured feedback including:
- Could core points be achieved within time limits
- Could recent story points be achieved
- Usability of interface
- Accuracy and applicability of naming, tips, etc.,
and some less structured:
- Problems noted
- General Impressions
- Suggested Improvements
Possibly this could be followed with a "Can you break it" session with questions of could you - how?
Usually you would look to have at minimum an experienced user, an inexperienced but technically aware use possibly a member of another team or a teenager, a totally inexperienced user if training is not normally provided consider a member of clerical or non-technical managerial staff for this and a typical customer end user.
You can consider allowing anonymous feedback but if so the user should indicate level of experience and background which for the short term will render it less anonymous in the short term but will allow weighting of the results with customer representative feedback on usability being given a very high weighting.
Version control your results/feedback with a reference to as your source code version information.
Also, consider using a screen recorder and possibly a video camera for supplementary information, the first so that if it turns out that clicking on the pixel at [12,42] crashes the app or does something strange you can reproduce the problem. The second will let you see when the user is having problems knowing what to do next even if they say that they had no problems.
Most important plan, structure, ensure that you can reproduce and aim to compare apples with apples not with mangoes.