Amazon
Alexa Conversations Description Language
* This work is protected under an NDA.
ROLE
UX Research Intern
TIMELINE
May 2021 - August 2021
GOAL
Inform CX decisions and improve the skill building experience with the ASK CLI and ACDL
METHODOLOGIES
Interviews, Observation, Survey, Thematic Analysis
During the summer of 2021, I interned at Amazon as a UX Researcher in the Alexa Conversations team for 12 weeks. Alexa Conversations (AC) is an AI-driven approach to providing more natural conversational experiences with Alexa. In my team, I learned about Alexa skills, which are a set of actions or tasks that are accomplished by Alexa to help customers perform everyday tasks or engage with content naturally through voice. As another way to create these skills with more flexibility and modularity, skill builders could use the Alexa Skills Kit Command Line Interface (ASK CLI) with Alexa Conversations Description Language (ACDL). As a beta feature, however, ACDL required more user research to help inform future design decisions and guide its development. For my project, I conducted a generative user research study to evaluate the experience of skill builders creating AC skills using the ASK CLI with ACDL.
Although my internship was virtual, I had the opportunity to conduct interviews for my research on-site in the Seattle office for two weeks. It was an amazing experience touring Amazon's headquarters and meeting my team in person.
Project Journey
During my internship, I met with stakeholders, created a research plan, recruited research participants, conducted contextual interviews, analyzed data, wrote a research report, and presented findings and recommendations to stakeholders. Throughout the process, I consulted the UX team and stakeholders for feedback and suggestions to ensure alignment on the goals of my project.
Planning
As I prepared for the research study, I wrote several drafts of the research plan. The plan included an overview, research questions, objectives, scope, an interview script, screening criteria, and a timeline. Parts of the plan that took much of my consideration were the research methodology and the interview script. After several iterations of the research plan, I designed the structure of the interview as the following:
-
A brief explanation of the study
-
Introductory questions about a participant's familiarity with AC and ACDL
-
An observation of how a participant built a skill
-
Questions about a participant's experience using the ASK CLI with ACDL
-
Key performance indicator (KPI) survey to provide a numerical evaluation of a participant's experience
Challenge
Choosing of research methodology. While writing the first draft of the plan, I initially thought an hour-long interview, in which a participant builds an AC skill from scratch and deploys the skill, would be the best method for evaluating skill builders' experience with using the ASK CLI with ACDL. However, I realized the task would take longer than an hour, and most participants would be unable to join an interview that lasts more than an hour because of various reasons (e.g., work). Due to the time constraint, I decided to go with a contextual interview: a participant would share their screen and walk through their experience with a skill that they have previously created.
Decreasing the number of interview questions. Writing the interview script took quite some time as well, because my stakeholders and I had many questions about the skill builders' experience. Cutting down the number of questions to fit the timeframe of the interview was a challenge. Eventually, I decided on the questions to keep by considering which of them would be essential to answering my research questions and achieving the objectives of the study.
Addressing conflicting feedback. As I wrote my research plan, my manager, mentor, team, and stakeholders reviewed each draft and provided feedback. At times, a suggestion from one person would conflict with another person's. To address a conflict, I would ask for further clarification about the feedback and the reasons behind it, if necessary. Keeping the research goals in mind, I made decisions about what revisions to add to my plan and consulted with those directly involved in the study (i.e., my manager and stakeholders) to align my decisions with them.
Recruitment
Following the completion of the research plan, I recruited Amazon employees (first-party, or 1P) and people external to Amazon (third-party, or 3P) as participants of user interviews via email and Slack. In Amazon's internal workspace and in various external Alexa workspaces on Slack, I posted a recruitment message on public channels and directly reached out to individuals that have used or potentially used ACDL. I also emailed people who did not have Slack, but have used or possibly knew others who have used ACDL.
Challenge
Recruiting the desired sample size with a diverse selection of participants. Recruiting 15 participants proved to be more difficult than I had thought. As the launch of the Alexa Live event was quickly approaching, most Alexa teams were busy with preparing for the event and were unable to participate in an interview. Though the recruitment process was tentatively scheduled to be two weeks long, I extended it to three weeks and was able to reach the desired number of 15 participants. On the third week, I was simultaneously recruiting and conducting interviews. Another obstacle that I faced was recruiting an approximately equal number of 1P and 3P participants to get a diverse set of data. Due to the ACDL being a beta feature, there were few 3P developers who used ACDL. Searching for people who mentioned ACDL on Slack and online, I reached out to all of them and was able to recruit a handful of 3P participants.
Interviews
After I recruited 15 participants, I conducted recorded interviews over the course of two weeks through Amazon Chime. During the interviews, I asked the participant my questions, observed them walk through their experience building an AC skill using the ASK CLI with ACDL, and took down notes all the while. At the end of the interview, participants completed a survey, rating their experience with ACDL.
Analysis
Once I completed my interviews, I used Excel to organize the data I collected into various categories that would answer the research questions in my study plan. For example, when I found a pain point, I counted the number of interviewees who shared the pain point and quoted their thoughts about the issue. This approach to organizing my data helped me rank the priority level of pain points and think of recommendations to address these issues.
Research Report
After analyzing the data that I collected, I consolidated my findings and recommendations into a 6-page research report. Including appendices of the data, the report was over 15 pages long. As I wrote the report, my manager, mentor, team, and stakeholders reviewed the report and provided feedback. Once I completed the report, I scheduled and facilitated a 1-hour final readout with stakeholders and Amazon employees who were interested in the results of my research. During my presentation, I reported my findings and aligned my recommendations with stakeholders.
During the two weeks that I was in Seattle, I conducted interviews in one of the lab rooms at Amazon's new Nitro North building. I really enjoyed using the lab's multi-camera and audio setup while I held my virtual interview sessions.
Outcome
At the end of internship, I submitted my final deliverables:
-
Research plan
-
Research report
-
Video recordings of interviews
-
Summary notes of interviews
-
Final presentation to stakeholders
Based on my research findings, current design decisions gained further support. Stakeholders added changes to the product roadmap of the ASK CLI and ACDL as well. Lastly, as a result of my recommendations, my team will conduct a follow-up study on ACDL. Due to the few 3P developers who have used ACDL, they plan to reach out to same 3P participants that I have found for future studies as well.
Reflection
Interning at Amazon was a major learning experience for me in the world of UX. As I was new to both the company and user research, I faced several challenges and overcame them with the support of my manager, mentor, and team. I consulted them for advice and actively sought feedback on my work numerous times. Coming out of this internship, I learned a lot about how to plan and conduct a research study.
Avoid leading questions. When I received feedback on my first draft of the research plan, I learned the term, "leading questions," and the downfalls of it. I realized that an interviewer's choice of words is important as it can affect a participant's feedback or behavior. In subsequent drafts, I made sure to consider the phrasing of my questions to avoid implying or including a desired answer.
Build rapport with a participant. When I wrote the interview script and conducted interviews, I learned about the importance of building rapport with a participant. By helping a participant feel comfortable and open as they speak, an interviewer can draw out authentic and candid feedback for improving the user experience of a product. Keeping this in mind, I structured my interview to start with a brief conversation to be more personable and then asked questions that were easy to answer. During the interview, I nodded, maintained eye contact (through the video camera), and gave verbal acknowledgements (e.g., "I see") to ensure the participant feels heard. Another factor that I considered was to avoid asking consecutive questions in one go to prevent participants from experiencing a cognitive load as they try to remember and answer all of the questions at once.
Prepare to answer the question "why" for all your research decisions. Throughout my internship, I was often asked why I decided to go with a certain research decision. For example, why did I decide to conduct 1-hour interviews? Why did I decide to recruit 15 participants—not 10 or 20? Eventually, I started to record my the reasons why I went with one research decision and why I did not go with other options in a notebook. This helped clarify my thoughts about the decisions I made and prepare me to answer the next "why" question.