IN A NUTSHELL:

Google is a company known for creating easy and intuitive interfaces. When tasked with assessing the usability of their 2nd generation Google Chromecast streaming device, specifically the ease and intuitiveness of the out-of-the-box experience, I expected to find a small number of relatively minor issues. Yes, we were testing with a specific user group (i.e. U.S. adults ages 45-60, who had limited experience setting up electronic devices, or who typically required support to do so). But still...

What my team of 4 and I discovered surprised us. Over the course of a group expert review and in-home usability study with 10 participants, we unearthed 64 issues, 15 of which were classified as "critical" (see "severity scale" in gallery above). In fact, none of the participants were able to complete the setup on their own (i.e. without assistance from an evaluator or a family member), and the process received a very low SUS score.

While some may have felt demoralized by these findings, I felt energized. Here was an opportunity to truly make an impact and improve the user experience created by a popular, and largely beloved, brand.

 

Accordingly, I spearheaded our efforts to generate cost-effective design recommendations for all 64 issues. 

Specifically there was a need for:

  • More comprehensive instructions, onboarding (e.g. presenting a tutorial video at the beginning of setup process), and technical support for non-tech savvy users (e.g. explaining what an HDMI port is and where it is located on a TV)

  • Greater visual clarity in packaging instructions and UI of the app and browser extension

  • Increased visual salience of various setup cues and casting icons

  • Greater user control at points in the setup and casting process

 

The group expert review and final usability testing reports can be downloaded here:

 

 

 

Group expert review        Usability Testing Report

MY ROLE:

  • Spearheaded drafting of research study proposal and test plan

  • Conducted an individual expert review of Chromecast out-of-the box experience

  • Wrote moderator's guide, including tasks, pre- and post-test interviews

  • Conducted in-home usability test with 2 participants

  • Presented expert review and usability test findings for phase 2 of the out-of-the-box experience (i.e. downloading the app or browser extension) to client

  • Drafted and presented summary of research results to client in written and oral form

  • Spearheaded generation of design recommendations

SKILLS INVOLVED:

  • Drafting a research study proposal

  • Drafting a usability test plan

  • Conducting an individual expert review

  • Consolidating findings in a group expert review

  • Creating a moderator's guide, including tasks and pre- and post-test interview questions

  • Recruiting participants

  • Screening participants via phone

  • Conducting in-home usability testing

  • Administering the System Usability Scale (SUS)

  • Analyzing expert review and usability test results

  • Presenting expert review and usability test results to client in written and oral form

Want more detail? Read on...

PROBLEM STATEMENT:

Google believes that users can set up their Chromecast device on their own, without external support. Their aim is to make the setup process fast and intuitive, ensuring a high level of product usability and a low likelihood of product returns. Google seeks to understand and evaluate the current setup process for a target user group of individuals between the ages of 45-80 who typically do not set up electronic devices in their home, or who often need help when setting up such devices.

STUDY GOALS:

  • Assess the ease and intuitiveness of the self-service setup process for the Chromecast device

  • Find points of user error and frustration

  • Determine opportunities for design improvements in both the hardware and software

SCOPE:

This research study is limited to the Chromecast setup process on iOS devices (i. e. iPhone, iPad, Mac laptop).

TARGET USER GROUP:

  • Does not to typically set up electronic devices in their home, or typically needs help when setting them up

  • 45-80 years old

  • Has access to an iPad (iOS 7.0 or higher), iPhone (iOS 7.0 or higher), or a Mac laptop (MAC OS X 10.7 or higher)

  • Has a TV with an HDMI port  

  • Has access to a home Wi-Fi network

  • Has Apple ID account and access to login info

  • No prior experience with Google Chromecast

METHODOLOGY:

Group Expert Review

  • Determined prioritization scheme (i.e. severity scales) based on client’s study and business goals. 

  • Identified target user group, based on client interest. 

  • Generated task scenarios and tasks, based on study and business goals. 

  • Individually assessed the setup process for the Chromecast from the perspective of our target user group.

  • Consolidated findings as a team, discussing duplicative issues and coming to consensus on finding descriptions, severity ratings, and categorizations. 

  • Grouped findings according to phase in the setup process.

Usability Testing

  • Recruited a total of 10 participants.

  • Participants were screened via an over-the-phone questionnaire.

  • Participants did not receive compensation for their participation in this study.

  • Each of the five evaluators moderated two 1-on-1 usability sessions. 

  • 10 one-hour usability sessions were held in-person, in the participants’ homes, over the course of two weeks (April 7-21, 2016).

  • Participants were asked to use their own electronic equipment to conduct the study including their home television set, home WiFi network, Apple ID credentials, and Chromecast-compatible Apple device of choice (i.e. Mac laptop, iPad, or iPhone).

  • Photos and audio were captured during the usability sessions.

PROCESS:

We began by assessing our stakeholders needs and goals for the research study. We then submitted a proposal to our client, articulating the problem statement, study goals and scope, target user group, methodology, assumptions, deliverables, and timeline. 

 

Once approved, we formulated a comprehensive test plan. The test plan provided specific details on our planned group expert review and usability test, including our recruitment plan, participant screener, tasks, analysis plan, and full moderator's guide. As with the proposal, the test plan was submitted to our client for feedback and approval. 

Following an in-depth discussion about the expert review process and categorization/prioritization schemes, my colleagues and I each completed an individual expert review on the Chromecast out-of-the-box experience. The review was conducted from the perspective of our target user group, taking into account factors such as age-related loss of visual acuity.

 

We then consolidated our findings as a team, discussed duplicative issues, and reached consensus on finding descriptions, severity ratings, and categorizations.

Following the expert review, we conducted in-home usability testing with 10 participants in our target user group, assessing whether they could complete the setup process without outside support, as well as when and where they made errors and expressed negative emotions (e.g. confusion, frustration).

Each participant completed:

  • A background questionnaire to verify their information

  • A pre-test interview to assess prior knowledge of the Chromecast device

  • A series of tasks, each of which was followed by a set of questions

  • System Usability Scale (SUS) test

  • A post-test interview

 

As with the group expert review, we consolidated our findings as a team, discussed duplicative issues, and came to a consensus on finding descriptions and severity ratings. 

 

We presented the findings from both the expert review and usability test to our client. Each finding was accompanied by a design recommendation. In addition to the presentation decks, we provided our client with written reports and a spreadsheet describing every issue encountered. The spreadsheet additionally listed the phase during which the issue was encountered, its severity, its category (e.g. navigation, visual clarity, user control), its location (e.g. packaging, software, hardware), the business goal impacted, and our design recommendation. 

TASKS FOR USABILITY TEST:

1) Assess the Chromecast box

“You just received a birthday gift in the mail from a friend. You unwrap it, revealing this box: ● What are your impressions of what this is and what it can do?

  • What do you think the setup steps are?

  • How long do you think it will take to set up?”

2) Follow instructions to setup the Chromecast

“You’re hosting the birthday party at your home later today and would like to set up the Chromecast before your guests arrive. You’re hoping to use it to play entertainment on your TV. Please set up the Chromecast as you would normally and let me know when you have completed the setup for the Chromecast, or have gone as far as you can. ”

 

3) Cast content from device using the Chromecast

“Next I would like you to play audio or visual content on your TV using the Chromecast. Let me know when you have completed the task, or have gone as far as you can.”

SUMMARY OF FINDINGS

 

Participants were excited to use the Chromecast and were impressed by its functionality. Several participants expressed an interest in purchasing one following the usability study.

 

However, the Chromecast setup process proved to be highly challenging for the target user group. None of the participants were able to complete the setup on their own (i.e. without assistance from an evaluator or a family member), which conflicts with a key business goal. Additionally, the setup process received a very low SUS score. If Google wishes to attract and retain customers in this user group, they must take steps to improve the ease and intuitiveness of the setup process.

 

In our findings, we identified areas where the product could better support this target user group, ensuring that they are able to complete the setup process without external support.

 

Specifically, there is a need for:

  • More comprehensive instructions, onboarding (e.g. presenting a tutorial video at the beginning of setup process), and technical support for non-tech savvy users (e.g. explaining what an HDMI port is and where it is located on a TV)

  • Greater visual clarity in packaging instructions and UI of the app and browser extension

  • Increased visual salience of various setup cues and casting icons

  • Greater user control at points in the setup and casting process

Google Chromecast

Conduct usability testing and an expert review of Google Chromecast out-of-the-box experience 

Chromecast Product Overview
Phases in Setup Process
Severity Scale
Sample Usability Test Finding
Sample Usability Test Finding
Sample Usability Test Finding
Sample Usability Test Finding
Sample Usability Test Finding
Sample Usability Test Finding
Sample Usability Test Finding
Sample Usability Test Finding

© 2017 Vanessa Wiegel