5

I am attempting to evaluate several audio codecs to determine their suitability to transcode speech for use with voice assistants such as Amazon Alexa and Google Assistant.

From my own research, there doesn't seem to be much, if any test samples available that could be used for this purpose. Especially for keyword detection such as "Alexa" or "Hey Google".

Ideally, I'm looking for samples that I can use that will return a deterministic response from the different assistants. For example: "What year was [X celebrity] born?"

Does anyone know if such a library of test samples exists? Either commercially or free to use.

Many thanks!

4
  • I can't find much either - you may have to create a suite for yourself... which might be a nice / sellable product at the end of it! Let us know how you get on.
    – dvniel
    Jun 19, 2018 at 7:33
  • 2
    Thanks for looking! Yeah I suspected that's what I'd have to end up doing. The hard part will be getting a broad enough spectrum of languages/accents to avoid bias.
    – user33384
    Jun 19, 2018 at 8:04
  • 1
    An interesting question, not deterministic but numberous words with audio samples in many languages can be found at en.wiktionary.org/wiki/Wiktionary:Main_Page Oct 17, 2018 at 15:18
  • This might be helpful - voice.mozilla.org/en/datasets
    – JAINAM
    Jul 16, 2020 at 10:49

1 Answer 1

0

Unfortunately, it's unlikely that you will find a library of test samples for deterministic response from voice assistants such as Alexa and Google Assistant. The reason for this is that the responses generated by these voice assistants are highly dependent on the natural language processing (NLP) algorithms and the data sets used to train them.

However, you can use the publicly available voice assistant APIs, such as Amazon Alexa Skills Kit (ASK) and Dialogflow (for Google Assistant), to create your own test samples. To create a test sample, you can write a script or a list of queries, then use the API to make the queries and record the responses.

Once you have created a set of test samples, you can use them to evaluate the different audio codecs. To do this, you will need to transcribe the audio files into text, then compare the transcribed text to the expected results to see how well the codecs preserve the speech quality.

Additionally, you may want to consider using a quality assessment tool, such as PESQ (Perceptual Evaluation of Speech Quality), to objectively measure the speech quality of the codecs.

  1. Amazon Alexa Skills Kit (ASK) API: https://developer.amazon.com/en-US/alexa/alexa-skills-kit
  2. Dialogflow API (for Google Assistant): https://dialogflow.com/
  3. PESQ (Perceptual Evaluation of Speech Quality) tool: http://www.itu.int/rec/T-REC-P.862/en

Note: PESQ is a standard tool used in the telecommunications industry, but it may not be freely available for download. You may need to purchase a license or find a third-party implementation that you can use.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.