In previous articles on my LinkedIn profile, I have presented ideas on developing Alexa Skills and Google Home Actions including scripting, writing requirements, MP3s, APIs, and Lead Gen. If you’re wondering what the heck an Alexa Skill is, please check out my Alexa 101 post here.

As we develop Alexa Skills, we create a need to test our creations, just as in any software development process. Amazon has created a method for developers to test their skills on virtual Echo devices, real Echo devices, on developer accounts and non-developer accounts.

The magic of testing on a developer account, specifically the account used to create the Alexa Skill, is that once the skill is loaded into the developer portal, and testing is enabled, the developer is able to test on all of his/her Echo devices. This is usually pretty seamless and intuitive. Before we submit for Amazon certification, we also want to test on other accounts and other devices outside of our developer account. For example, we have a client that wants to test the skill on their Echo devices, or our product manager, or peer review, or we’re running a beta with friendly clients, etc.

Let’s start with the Beta testing documentation provided by Amazon. As an introduction, Amazon provides a summary blog post here. Next, you’ll want to review the official Amazon documentation here.

In developing, testing, and submitting numerous skills, here are a few lessons I learned the hard way about Beta testing Alexa Skills.

1.    Your Beta tester must ‘accept‘ your invitation in order to start testing. This means that they must click on a link inviting them to test the skill, click “Test Skill” button, and subsequently, “Enable” button. This sounds and in 90% of cases is intuitive to your tester, but ensure this step is complete before moving on. As a best practice, I always, invite the user to the Beta in the Amazon Developer Portal, then follow-up with a personal email with the “Invite URL.” One day later, I will check the Beta test portal and confirm that they have accepted my Beta test invitation. I may call or follow-up with an email to remind the tester to accept the invitation.

Why OH Why?

a.    Your Beta tester might not get the automated email from Amazon. Entering the tester email address in the portal should trigger an automated email to the tester. Keywords: should and automated. I’ve personally found the failure rate of automated emails extremely high.

b.    You might be using the wrong email address. As stated in the documentation, you must invite the email address associated with the tester’s Amazon Alexa account. Some people use multiple email addresses and might have a separate email address for their Amazon account.

c.     Tester must be logged into their Amazon Alexa account when they accept your invitation. If they are logged into a different Amazon Alexa account, they will receive an error message, need to log out and back in with the correct account and then start the accept URL process again.

d.    If your beta tester has a device with 2 accounts, ensure that they are on the account that they have accepted your Beta invite. Have them say, “Alexa, switch accounts” if they have accepted your invitation and still cannot launch the skill.

Same Name Skills

2. If there is already another skill with the same name approved in the Alexa Skill store, Alexa may automatically open that skill, and not the one you’ve just accepted the Beta test. For example, I had a client who wanted to name their company skill, ‘StoryTime.’ There was already an unrelated and approved Alexa Skill with the same name. My testers were able to successfully accept the Beta invitation, but when they tried to launch the skill on their Echo devices, they would launch the production ‘StoryTime’ skill instead of the test ‘StoryTime’ skill. My testers had never enabled the original ‘StoryTime’ skill, but according to CNET as of April 2017, Alexa will allow you to launch a skill without enabling it first.

Additionally, If the tester has another skill of the same name in development, Alexa will automatically open that skill instead of the Beta. #TrueStory I attended an Alexa Dev Day workshop with Steve Youngblood, and Steve came up with the idea for a ‘Lawn Advisor’ skill. As a workshop exercise, we both created ‘Lawn Advisor’ skills into our individual Amazon Developer accounts. A few weeks later, Steve invites me to his Beta test; I accept and when I say, “Alexa, launch Lawn Advisor” my Echo pulls the Skill I created on my account in the workshop (very incomplete), instead of Steve’s Beta Skill. Bottom Line: As of the writing of this article, if you have multiple skills enabled (or not enabled) with the same name, Alexa does not prompt you to ask you which skill you want, she will simply pick one which may or may not be the one you intended.

International Testers

3.    Beta testing can be tricky with international testers. If your skill only has language model for the US, your UK tester can accept your Beta test invitation, but their Echo device may not necessarily be able to actually test your skill. To get this to work, you will need to add an English (UK) interaction model to your Alexa Skill. You should also consider adding an additional endpoint to lower latency in Ireland in AWS.

I had one client with a UK address activate their Amazon Echo device in the UK, and then got on a plane and flew to Morocco. This client then connected his Echo to the Wi-Fi in his hotel in Morocco, and the Beta skill worked seamlessly.

Find Good Testers

4.    Find good testers. Ok, I know that sounds rhetorical, but trust me. It is not useful to get feedback such as, “Uh, it didn’t work.”

Ideally find testers who are savvy enough to make an audio recording of themselves testing the skill. Ask your testers to make a recording of themselves with an app such as Voice Recorder and have them upload to Slack, Dropbox, or even email it to you.

Depending on your test plan, you may (or may not) want to create detailed test instructions for your testers. Perhaps, you want to conduct User Acceptance Testing to ensure your Alexa Skill works and pulls the appropriate APIs, or perhaps you want to know how intuitive your voice prompts are in getting the user to their final destination? Either way, you’ll need to help them enable your Beta Test Alexa Skill.

Next, we’ll discuss best practices for getting your skill certified and approved by Amazon for public distribution.

In the meantime, check out my latest Alexa Skill that provides median rental prices from Zillow based on your zip code. Enable it here -> https://amzn.to/2Rz3g5t and then say, “Alexa, open Rental Prices”