IN A NUTSHELL:
Google is a company known for creating easy and intuitive interfaces. When tasked with assessing the usability of their 2nd generation Google Chromecast streaming device, specifically the ease and intuitiveness of the out-of-the-box experience, I expected to find a small number of relatively minor issues. Yes, we were testing with a specific user group (i.e. U.S. adults ages 45-60, who had limited experience setting up electronic devices, or who typically required support to do so). But still...
What we discovered surprised us. Over the course of a group expert review and in-home usability study, my team of 4 and I unearthed 64 issues, 15 of which were classified as "critical" (i.e. caused significant confusion and/or frustration; may necessitate an assist from a third-party (e.g. from customer support, friend/family, evaluator); may prevent task completion; had a significant impact on Google's business goals).
Google believed that users could set up their 2nd generation Chromecast devices on their own, without external support (i.e. Chromecast customer support, third-party support). Their goal was for the setup process fast and intuitive, ensuring a high level of product usability and a low likelihood of product returns.
My team of 4 and I sought to understand and evaluate the current setup process, including if and when a user needed to seek out external support. Our target user group was U.S. adults ages 45-60, who had limited experience setting up electronic devices, or who typically required support to do so.
We each completed an individual expert review from the perspective of our target user group. We then consolidated our findings as a team, discussed duplicative issues and reached consensus on finding descriptions, severity ratings, and categorizations.
Additionally, we conducted in-home usability testing with 10 participants in our target user group, assessing whether they could complete the setup process without outside support, as well as when and where they made errors and expressed negative emotions (e.g. confusion, frustration).
Each participant completed:
A background questionnaire to verify their information
A pre-test interview to assess prior knowledge of the Chromecast device
A series of tasks (described in greater detail on next slide), each of which was followed by a set of questions
System Usability Scale (SUS) test
A post-test interview
We consolidated our findings as a team, discussing duplicative issues, and coming to a consensus on finding descriptions and severity ratings.
We also identified opportunities for design improvements based on the usability testing results and a group expert review.
Conduct usability testing and expert review of Google Chromecast out-of-the-box experience
Want more detail? Read on...