Problem/Opportunity Overview

As a top game streaming service, Twitch has millions of users watching streams every day. It is essential that its chat system is inclusive and provides an optimal experience for all users.

A significant benefit is that the channel community on Twitch allows everyone to have an equal opportunity to contribute to the conversation in chat. However,  chats in popular live streams tend to move rapidly with increasing viewers participating in the discussion, which leads to the chat overloading with different messages. This behavior denies the opportunity for viewers to have a meaningful conversation with the streamer. As a result, chat participants disengage from the streaming service.

Phase 1: Research

Survey Distribution

Objectives:‍

• Understand Twitch users' goals and pain points while reading the chat.

• Recruit participants for interviews ‍

‍Analysis

We received 49 responses within a week of distribution. Out of the 49 respondents, 47 identified as Twitch users, we only counted their responses for the rest of our analysis.

Findings

• More than 50% of respondents selected that the chat was the service they use the most.

• Participants stated that the chat goes too fast and they are not able to read anything.

• More than half of the respondents identified as Twitch viewers.

Semi-Structured Interview

Objectives:

To further understand users’ reading difficulties, as discussed in their survey responses, asked interviewees about their experiences having difficulties reading Twitch chat‍

Analysis

We entered our interview notes and transcripts into a spreadsheet. Then, we added them to Miro and created an affinity diagram. Individually, we walked the wall, and we came back together to discuss further implications. See below our diagram. ​​​​​​​​​​​​​​

Findings​​​​​​​

We identified the following themes from our user interviews:

Phase 2: Design Implications

Based on our research findings and insights from phase 1, we developed the design implications for our future prototype including our user persona and empathy map.

‍

Phase 3: Design

Feature 1: Chat Settings Redesign

Our initial research showed that users were unaware of the existing customization capabilities that Twitch provided for the chat view. The font customization feature is currently hidden in the chat feature of Twitch, and we restructured the UI so that the chat setting is at the primary level.

Additionally, having a clear chat setting provides a centralized place where users can easily discover other chat functionalities. To increase legibility, we also included the ability to change font style, to better accommodate readers who may have a harder time reading certain fonts.

Feature 2: Chat Playback Speed

Our research showed that users had varying experiences keeping up with the Twitch chat feeds based on which channels they followed.

Chats of channels with fewer viewers and chat participants were easier to keep up with, however, chats of channels with a large number of viewers were difficult to read. By having a speed customization feature within each channel chat, the user can customize speed settings based on each channel.

When doing usability testing, we noticed that it was difficult for some users to navigate directly to the chat playback speed so we moved it over to the main chat settings page for easier access. By adding this option to the main chat settings screen, we aim to assist users in saving time to change the playback speed of their chat.

Feature 3: Chat Highlights: Understand what the general chat is saying

Users emphasized wanting to understand the general gist of chat in a manner that took a short amount of time. To address this want, we created a new feature to add to the Twitch chat called CrowdSpeak AI. This functionality took what users were saying in chat and described it in a short blurb that was visible to all users viewing the chat. For example, if users were talking about how to slay a certain mob in Minecraft multiple times in a short time span, the chat highlighting functionality powered by an AI would summarize in a short line of text for users to quickly scan and understand what is going on.

When “Chat Summary” is expanded, the messages that helped the AI create the summary are shown in a list based when they were said in chat. In future iterations, we would add a search bar functionality that allows users to search for specific keywords mentioned in the chat.

Phase 4: Design Evaluation

Expert Evaluation

Objective: to uncover the common usability issues that experts can quickly identify following the heuristic principles.

Rationale: Expert heuristic evaluation helped us identify usability problems that non-experts might not notice.

Method:

• Three Experts evaluated our prototype according to Nielsen Norman’s 10 Heuristics.

• We asked experts to think aloud while going through our prototype and we asked follow-up questions.

Analysis

To analyze the heuristic evaluation, we grouped the three expert responses into a spreadsheet to analyze and compare each expert’s scores and comments.

Findings

• The three experts agreed that there were cosmetic issues with our prototype but should be given less priority.

• Experts suggested that giving the users to ability to preview the setting could improve our visibility of system status.

Usability Testing

Objective: To test whether we were creating a user interface that is simple, intuitive, and easy to understand for our target user group as well as meeting their needs.

Rationale: we wanted to test our prototype with Twitch users to gather feedback from our target population.

Method

• We used a task-based usability test with embedded semi-structured interviews after each task.

• Utilized Figma to display the prototype to participants, allowing them to click through and interact with the prototype on their own.

• Used WebEx to conduct virtual usability tests.

• Asked participants to fill out a System Usability Scale (SUS) survey once they finished all the tasks.

Analysis

• Analyzed the SUS scores of each participant following the SUS guidelines

• Organized the notes from our usability study to Google Sheets, and then we transferred to a Miro Board.

• Divided the notes by each task, and we collectively worked to find common themes that participants brought up during the session.

Findings

• The average score from all participants was 71 which is close to the average for a system that is likely to be recommended to use by others according to the SUS guidelines.​​​​​​​

• We found a common issue that participants had with the information architecture of our design.

Recommendations  

After conducting usability sessions and expert evaluations, we were able to draw significant

conclusions about how the design of our prototype should change going into further iterations.

We identified three key issues that our design would need to address. We received great information that allowed us to look at our prototype differently. ​​​​​​​

Watch our final presentation to learn about our work!