How Speech Training Should be
Re-present
Represent is an award-winning/proven-effective solution that aims for improving student's speech skills through Virtual Reality(VR).
Currently, it has been used by Carnegie Mellon University community for 5 semesters, over 98% of the users believe it is helpful for improving their speech skills.
Problem
Improving speech doesn't need to be awkward and boring
Solution
Designed and implemented a VR Training App on Oculus. Implemented into CMU course that help students improve their speech skills through system feedback and self-reflection.
Impact
Gelped 200+ students improve presentation skills in 5 semesters. App has 98% satisfaction rate and it's affiliated with CMU Teaching Innovation Award
My Role
Lead Product Designer, UX Researcher, 3D ArtistContext
Speech Training in VR
Re-Present helps people enhance their speaking skills by providing features where they can practice and review their performance.
Problem Statement
Improving speech doesn't need to be awkward and boring
Issues in speech and practicing process
Feeling anxious to give a speech
Fear for the potential
failure and errors
Awkward speech practice process
Lack of motivation and guidance
Unaware of
their
pros and cons
Competitive product missed out opportunities
* * *
Research - how do you know it's a human problem
Gathering Inputs from as Many Sources as Possible
How to evaluate speech?
Academic/professional standards
Our client, Kim Hyatt, is a public speaking expert and the professor for the course "Strategic Presentation Skills" at CMU, she made it clear that the VR App is intended for her course use. Therefore it should be evaluated by an academic standard.
Multiple factors to evaluate
Through the course Evaluation Rubric, related papers, and course recordings that the client provided, we realized that the performance could be affected by Time, Pace, Flow, Volume, Vocal Expressions, Eye Contact, Body Language, Message Effectiveness, and so on... and we also found that students are likely to ignore some of these while presenting.
Interview: what's wrong with the existing ways of practicing?
Recording or practice in front of the mirror
Through semi-structured interviews with the course TA and 10+ CMU students, we found that students typically record themselves or practice in front of the mirror before giving a speech.
Lack of awareness of their strengths and weaknesses
We also found that there was a general lack of self-awareness and opportunities for self-observation. One interviewee commented, "I don't really know what I'm doing wrong".
Synthesize three typical target users
Glossophobia - Feel anxious in public speaking
people of all ages and occupations struggle with public speaking—77% of the US population feels some anxiety with public speaking
1. Poor at presentation
2. Feel anxious before and during the presentation
Script Memorizer - Just practice but not sure about their pros and cons
Recite the script and record themselves or practice in front of someone
1. Recite the script
2. Not sure about pros and cons
Orator -who is eloquent or skilled in public speaking
Type 3: Never practice and don’t have much motivation in improving.
1. Demotivate to practice more like improv.
2. Believe there are not more to improve
Competitive Testing & Analysis
We tested the existing VR applications on ourselves and our peers, we identified several opportunities from those Apps:
1. The emphasis on object evaluation through summary report were provided with little to no context.
2. The user experience of going through each practice cycle felt too tedious, often aggravated by ambiguous UI elements and pages that slowed down the process.
Research Insight Summary
“Feel anxious to give a speech”
“Fear for the potential failure and errors ”
“Lack of motivation and guidance”
“Existing product experience is not ideal, blue sea there”
From Learning to Ideation
Lead team ideation through design sprints and “Crazy 8” storyboarding
Leading the Design Sprint and Crazy 8 method for the team to flush out ideas.
Solution Overview
Represent
Maximize your confidence in speech through
VR practicing, machine generated feedback, and self-reflection
* * *
Design Principles
Motiviating
Our App should empower the user to see speech practice as a growth opportunity than an obstacle.
Enlightening
Our App should shed new light on user's strengths and weaknesses of their presentation.
Effortless
Our App should transform the practice loop into an effortless process.Design a motivating / reinforcing loop
How we end up designing an experience with practice and review mode? Why save for recording? why design real-world-like settings? How to work with the course outline?
Framework: An Iterative Feedback Loop
We wanted to design a framework that people can gain confidence every time, so high-level speaking, “Reinforcing feedback“ loop is necessary.
Practice-Review Iteration Cycle
Practice Mode
In this experience, there are three big fragments: There is a Practice mode, a space that provides a classroom setting where users can upload a presentation material as a pdf format and practice their presentation with an attentive audience.
Practice Summary
Once they have finished the Practice mode, the following is a summary page where the users can see their performance results. It includes the number of filler words and the level of eye contact with the audience members. The Summary page is provided before the users enter the Review mode, in order to guide them on what components to focus on.
Review Mode
Now they enter the Review mode, where they become one of the audience members and review their performance. A timeline tool is provided on the virtual desk to enable users to navigate between different moments.
* * *
Shed light on the pros and cons for the speaker
Providing insights to user’s speech performance, and let them know that where to improve.
Provide post-practice feedback and insight into their speech
Before the user enters the Review mode, it was important that the user skim over a summary of their performance data to provide a general idea and set expectations for the experience.
The data was initially intended to be a concrete evaluation filtered by the system, however it was determined that an “indicative” data than an “evaluative” one would prove to be more informative so that the user can ultimately judge themselves.
Eye-gaze heat map and Eye Contact UI
Track Behaviors and Highlighting Noticeable Moments
Since we’re tracking data like user’s movement, head movement, audio, frequency of viewing PDF screen and Audience engagement rate.
We have a concept to “Highlight“ the noticeable moment for user , by analyzing these data and map out on the timeline.
With this visualization design, we hope that we could pinpoint where is the issue in the recording and serve as a virtual “coach“ to provide gentle idea to let user self-reflect. Although it’s desired features, it seems like adding 3-6 months to the scope, so we backlog it for the future development team.
* * *
Build a comfortable and easy-to-use experience in VR
Crafting a comfortable environment and user-friendly UI / interaction is key for every Virtual Reality Experience. As a lead designer, I put a fair amount effort in the User Experience.
#1. UX Design in Practice Mode
Account Logins and “Landing Room”
A Landing page is provided to let user login with the school credential, they’re able to choose different mode to start or check their previous recordings and reflect back.
A virtual room is intentional design to provide a sense of virtual room instead of directly immerse user into the classroom and the audience.
Align VR settings to the real-world room
The goal for practice mode is to create a realistic environment where the student feels like they’re presenting to a real audience. We want to ensure the user’s presence during the VR environment, familiarity with the practicing environment, and comfort while practicing the presentation. The physical environment is simulating the scale and layout of the client’s classroom, in this way, the users could feel more familiar and easy to focus on their presentation just as usual. In addition, we added a confidence screen for them to use as the point of reference for their slides and time. To avoid the user getting out of the Kinect detect area, we set up a stage with different materials that indirectly controls them to stay in the play zone. We also invested time in adjusting the color and light settings. We made them in a warm and vibrant tune to satisfy our goal of motivating the users.
Joystick interaction design spec
As the major control of the entire experience, the joystick’s function is limited to provide an easy-to-use interaction.
Cursor Interaction & Haptic Feedback
We decided to include three types of feedback in the virtual experience:
1. Cursor’s interaction & feedback based on the distance, this is important to provide a better visual indicator of the focal point.
2. Haptic feedback of Joystick is designed to provide feedback when users interact with interactive elements in the experience.
3. UI elements hover feedback, the hover feedback provides better understanding for the user about where they are currently focusing.
Audience Design
-Initially, the audience members were designed to provide some implicit real-time feedback based on the user’s performance, such as yawning or slumping back into the chair if the pacing was too slow. However, designing an evaluation system that seemed “fair” to the users seemed difficult, and the feedback was unnecessarily distracting the user from the presentation itself. Hence audience was designed to display only neutral, random idle behaviours that evoke a sense of realism, but not more.
UX Design in Review Mode
Once a user enters the Review mode, the user now becomes one of the audience members and watch oneself as a wooden figure. The gaze cone is added to the figure's head in order to highlight where the user was looking at. There are also floating color balls spread out in the classroom which visualize proxy eye contact level for each audience member. The user can also hear one's voice with a volume bar on the left side, which shows current volume with a average volume mark. A Timeline is provided to help a user better review one's presentation, allowing the user to play, pause, and scrub with a similar fashion when one uses a general video player. It enables the user to focus on specific moments and to navigate between different moments.
User’s Avatar Design
Why? People don’t like their own faces. We intentionally avoid uncanny valley effect for the user.
However, we need to present full picture of user’s body movement, so it ends up with somewhere in between.
Timeline UI Design
Since Timeline allows play, pause, and scrub features, the interaction involves not just a Timeline body but also controller and button interactions and interfaces. At first, we used a wand as a visual for the pointer interface that comes out from the index finger of the virtual right-hand model. However, Some people felt detached from virtual-self since the hand model was mannish and had a flesh texture and the hands don't match with their hands. Also, users were confused when they interact with the objects because there was no feedback when a user hover on to it.
To fix such issues, we add a virtual controller on the right hand and a virtual hand on the left hand. The user can switch the virtual controller and the hand as desire by clicking on the joy stick on the Oculus Touch controller. The rim light shader is used to the hand model to avoid the VR self-detachment.
Interaction
The pointer visual has changed into a laser that is coming out from the virtual controller model. This makes interaction more intuitive since the users can make interactions by using it as they use an actual pointer in the real life. Hovering state is also added where the laser is highlighted when it intersects with an interactable object.
Later, we added a small hit point at the intersection in order to clearly visualize where the user is hitting with the laser. Play and pause buttons are combined into one to make clear which button is interactable. Through out several iteration including polishing timing, size, and color, we were able to create intuitive interactions.
* * *
Iteration Based on User Testing
We have conducted user testing frequently and consistently throughout development; we were able to engage closely with our target audience from the beginning to the end of the product development cycle. We had direct access to our target demographic, especially non-native English speakers since a significant portion of the class cohort were international students.
Initially, we playtested each feature separately to validate the design questions relevant to that feature. And as the individual features were integrated into the system the playtest sessions became longer and longer.
Demo Video
Onsite Implementation
At the end of this project, we visited the Heinz school to set up the system ready for use. We installed and deployed in the VR training room
Along with the documentation, we hosted training sessions with Kim’s Teaching Assistants to ensure that they had a comprehensive knowledge of the system as the facilitators/supervisors overseeing the students.
Currently, the Re-Present system is being used in Professor Hyatt’s courses in strategic presentation.
Outcome
Implemented 5
semesters in CMU so far
Implementation - Re-present start being used in courses STRATEGIC PRESENTATION, 90-718, fro CMU 2019, Spring, Pittsburgh.
⬆Course Satisfaction Rate
We observed that the course satisfaction rate is increased comparing to the previous years.
Helped 200+ students
Starting from Spring 2020, they Represent had been supporting the course for more than 4 semesters, over 200+ students have used the system to improve speech.
*CMU Teaching & Innovation Award
Our client, Kim Hyatt was awarded the Carnegie Mellon University’s Teaching Innovation Award of 2020, with recognition of this system and its proven contribution to the CMU education innovation.
98% of user think it’s helpful
Through the initial pilot study, our client shared with us that there are 98% of user believes the system is helpful for their presentation and speech.
Research Paper Published
The performance data and results from Kim’s class are published in research papers.
What People Are Saying
“It is going to have an enormous impact and help many students and faculty, and it demonstrates the importance we at CMU place on developing the student as a complete individual — excellent in their discipline of study and outstanding at communication.”
— Ramayya Krishnan, Dean of the Heinz College, Carnegie Mellon University
“The Re-Present team went above and beyond expectationsto develop a VR application for my graduate course, Strategic Presentation Skills, at CMU. Based on the integration of this app and the type of data collected, I was the recipient of the 2020 CMU Teaching Innovation Award...the app proved to be an incredibly successful instructional tool with data to support continued research in this area.”
— Kim Hyatt, Associate Professor for course 90-178, Carnegie Mellon University
“Kim’s system (Re-present) has immediate implications for classes beyond her own discipline, In any field where students must present their work, they can benefit from an opportunity to improve their performance, Hyatt’s system (Re-present) also offers the opportunity for critical research on pedagogy.
— Jessica Hammer, Assistant Professor at Human-Computer Interaction Institute, Carnegie Mellon University
"With this technology, the course transforms from just a required course to an exciting chance to try something new. Because students receive instant feedback, they can begin to improve immediately, such as improving eye contact and lessening their filler words. I hear the students gain confidence each time they practice their speeches."
— Andreya Cherry, Teaching Assistant for course 90-178, Carnegie Mellon University