Multimodal Subtitle, Sensory Translation
Multimodal Subtitle, Sensory Translation
Subtitles that let hearing-impaired viewers
immerse in the movie’s emotions & context
Subtitles that let hearing-impaired viewers
immerse in the movie’s emotions & context
Subtitles that let Hearing Impaired viewers immerse in the movie’s emotions & context
Why I Care About Accessibility
Why I Care About Accessibility
Why I Care About Accessibility
Struggling to Grasp Movie Context During Temporary Hearing Loss.
Struggling to Grasp Movie Context During Temporary Hearing Loss.
Struggling to Grasp Movie Context During Temporary Hearing Loss.
I once experienced sudden hearing loss and felt the discomfort of missing everyday sounds. Since I love movies, I tried watching them with subtitles, but I couldn’t fully immerse myself. This naturally led me to think about the movie-watching experience of deaf viewers and inspired this project.
I once experienced sudden hearing loss and felt the discomfort of missing everyday sounds. Since I love movies, I tried watching them with subtitles, but I couldn’t fully immerse myself. This naturally led me to think about the movie-watching experience of deaf viewers and inspired this project.
Problem
Problem
Problem
Current subtitles failed to convey enough emotion and context.
Current subtitles failed to convey enough emotion and context.
Current subtitles failed to convey enough emotion and context.
Hearing-impaired viewers find it hard to immerse because subtitles don’t fully convey emotions, context, and sound storytelling, making it difficult to enjoy movies on the same level as hearing viewers.
Hearing-impaired viewers find it hard to immerse because subtitles don’t fully convey emotions, context, and sound storytelling, making it difficult to enjoy movies on the same level as hearing viewers.
Solution
Solution
Solution
Filling in missing sounds with visual captions, haptics, and vibration.
Filling in missing sounds with visual captions, haptics, and vibration.
Filling in missing sounds with visual captions, haptics, and vibration.
With subtitle presets and haptic feedback, the experience went beyond basic understanding to supplement emotion and context, enabling them to enjoy movies on the same level as hearing viewers.
With subtitle presets and haptic feedback, the experience went beyond basic understanding to supplement emotion and context, enabling them to enjoy movies on the same level as hearing viewers.
Current Streaming Platforms
Current Streaming Platforms
Current Streaming Platforms
Streaming platform subtitles often fail to convey background music or the actors’ emotions.
Streaming platform subtitles often fail to convey background music or the actors’ emotions.
Streaming platform subtitles often fail to convey background music or the actors’ emotions.
I’m okay
I’m okay
I’m okay
I didn’t realize the male lead was so nervous when he confessed.
I didn’t realize the male lead was so nervous when he confessed.
I didn’t realize the male lead was so nervous when he confessed.
Emotions from the actor’s voice
are not conveyed
Emotions from the actor’s voice
are not conveyed
Emotions from the actor’s voice
are not conveyed
( The victim felt fear )
( The victim felt fear )
( The victim felt fear )
Actually, they said a gunshot was heard in the background.
Actually, they said a gunshot was heard in the background.
Actually, they said a gunshot was heard in the background.
If the camera isn’t on the sound source, the sound is hard to understand
If the camera isn’t on the sound source, the sound is hard to understand
If the camera isn’t on the sound source, the sound is hard to understand
Interview Goal
Interview Goal
Interview Goal
How do gaps in subtitles affect hearing-impaired viewers
fully immerse in and enjoy the emotional journey of a movie?
How do gaps in subtitles affect hearing-impaired viewers
fully immerse in and enjoy the emotional journey of a movie?
How do gaps in subtitles affect hard of hearing viewers fully immerse in and enjoy the emotional journey of a movie?
Interview Outline
Topic
How do gaps in subtitles affect immersion and emotional experience for deaf and hard-of-hearing viewers?
Method
Mixed: Google Meet (online) + in-person
Participants: 4 HOH viewers
Questions on whether
Standard subtitles miss important elements
Gaps affect immersion and emotional understanding
Improved subtitles change the viewing experience
The amount of information feels overwhelming
Interview Outline
Topic
How do gaps in subtitles affect immersion and emotional experience for deaf and hard-of-hearing viewers?
Method
Mixed: Google Meet (online) + in-person
Participants: 4 HOH viewers
Questions on whether
Standard subtitles miss important elements
Gaps affect immersion and emotional understanding
Improved subtitles change the viewing experience
The amount of information feels overwhelming
Interview Insight
Interview Insight
Interview Insight
Unwanted subtitle spoilers reduce enjoyment.
Unwanted subtitle spoilers reduce enjoyment.
( John is running )
( John is running )
What, I got spoiled!
What, I got spoiled!
Limited sound immersion
weakens emotional impact.
Limited sound immersion
weakens emotional impact.
So that was the sound of cards? I didn’t know.
So that was the sound of cards? I didn’t know.
Need to ask someone when
not understood in the middle.
Need to ask someone when
not understood in the middle.
Madness is the only freedom left.
Madness is the only freedom left.
What did he just say? 😥
What did he just say? 😥
The hearing-impaired want the same level of understanding as non-disabled users.
The hearing-impaired want the same level of understanding as non-disabled users.
The hearing-impaired want the same level of understanding as non-disabled users.
“I want to be surprised, cry, and laugh at the same moments and directions.”
“I want to be surprised, cry, and laugh at the same moments and directions.”
“I want to be surprised, cry, and laugh at the same moments and directions.”
Current subtitles make it hard to enjoy the movie while following the content.
Current subtitles make it hard to enjoy the movie while following the content.
Current subtitles make it hard to enjoy the movie while following the content.
“I need to be in the right mindset to watch a movie.”
“I need to be in the right mindset to watch a movie.”
“I need to be in the right mindset to watch a movie.”
Literature Review
Literature Review
Literature Review
Visual, customizable subtitles and haptic make movies clearer for hearing-impaired users.
Visual, customizable subtitles and haptic make movies clearer for hearing-impaired users.
Visual, customizable subtitles and haptic make movies clearer for hearing-impaired users.
Font
Font
Visual Subtitles
Visual Subtitles
For music, visual expressions such as color, font, and position are more effective than plain text.
For music, visual expressions such as color, font, and position are more effective than plain text.
Ahn, S., Kim, J., Shin, C., & Hong, J.-H. (2024). Visualizing Speech Styles in Captions for Deaf and Hard-of-Hearing Viewers. International Journal of Human-Computer Studies, 103386–103386. https://doi.org/10.1016/j.ijhcs.2024.103386
Ahn, S., Kim, J., Shin, C., & Hong, J.-H. (2024). Visualizing Speech Styles in Captions for Deaf and Hard-of-Hearing Viewers. International Journal of Human-Computer Studies, 103386–103386. https://doi.org/10.1016/j.ijhcs.2024.103386
Customizable Subtitles
Customizable
Subtitles
Customizable Subtitles
Deaf & hard-of-hearing users prefer customization, and current subtitles lack representations of sound, emotion & distinction.
Deaf & hard-of-hearing users prefer customization, and current subtitles lack representations of sound, emotion & distinction.
Uzzo, G. (2025). “Your subtitles will look like this”: Exploring user preferences for closed captions across streaming platforms. The International Journal of Translation and Interpreting Research, 17(2), 55–72. https://doi.org/10.12807/ti.117202.2025.a04
Uzzo, G. (2025). “Your subtitles will look like this”: Exploring user preferences for closed captions across streaming platforms. The International Journal of Translation and Interpreting Research, 17(2), 55–72. https://doi.org/10.12807/ti.117202.2025.a04
Haptic Feedback
Haptic Feedback
Deaf & hard-of-hearing users can perceive sound information through vibration, so vibration assistive devices are useful.
Deaf & hard-of-hearing users can perceive sound information through vibration, so vibration assistive devices are useful.
Mirzaei, M., Kán, P., & Kaufmann, H. (2021). Effects of Using Vibrotactile Feedback on Sound Localization by Deaf and Hard-of-Hearing People in Virtual Environments. Electronics, 10(22), 2794. https://doi.org/10.3390/electronics10222794
Mirzaei, M., Kán, P., & Kaufmann, H. (2021). Effects of Using Vibrotactile Feedback on Sound Localization by Deaf and Hard-of-Hearing People in Virtual Environments. Electronics, 10(22), 2794. https://doi.org/10.3390/electronics10222794
Initial Ideation
Initial Ideation
Initial Ideation
Ideated research-based methods for intuitive understanding of movie sounds.
Ideated research-based methods for intuitive understanding of movie sounds.
Ideated research-based methods for intuitive understanding of movie sounds.
Question
Question
Question
From the hearing-impaired user perspective, I wondered
if too much information might disrupt immersion.
From the hearing-impaired user perspective, I wondered
if too much information might disrupt immersion.
From the hearing-impaired user perspective, I wondered
if too much information might disrupt immersion.
Codesign Workshop
Codesign Workshop
Codesign Workshop
Goal
Goal
Goal
Finding balance that helps hearing-impaired viewers
understand emotions & context without breaking immersion.
Finding balance that helps hearing-impaired viewers
understand emotions & context without breaking immersion.
Finding balance that helps HOH viewers understand emotions & context without breaking immersion.
Immersion
Immersion
Immersion
Rich understanding
Rich understanding
Rich understanding
Participants: 6
Duration: 90 minutes each
Format:
Co-design Workshop
Process:
0. Icebreaking
Identify easily overlooked sounds
Empathy mapping for discomfort points
Assess specificity in expressions
Test expression methods
Gather preferences for descriptive subtitles
Participants: 6
Duration: 90 minutes each
Format:
Co-design Workshop
Process:
0. Icebreaking
Identify easily overlooked sounds
Empathy mapping for discomfort points
Assess specificity in expressions
Test expression methods
Gather preferences for descriptive subtitles
*Permission to use interviewee photos in this portfolio has been obtained.
*Permission to use interviewee photos in this portfolio has been obtained.
Current subtitles score : 1.9
Current subtitles score : 1.9
Current subtitles score : 1.9
Q. How would you rate your satisfaction with watching movies on OTT platforms?
Q. How would you rate your satisfaction with watching movies on OTT platforms?
Q. How would you rate your satisfaction with watching movies on OTT platforms?
Strongly dissatisfied
Strongly dissatisfied
Strongly dissatisfied
1
1
1
2
2
2
Neutral
Neutral
Neutral
3
3
3
Satisfied
Satisfied
Satisfied
Dissatisfied
Dissatisfied
Dissatisfied
4
4
4
5
5
5
Very Satisfied
Very Satisfied
Very Satisfied
Don’t
Don’t
Don’t
Distracting!
Distracting!
Distracting!
Subtitles are positioned where the speaker is located
Subtitles are positioned where the speaker is located
Subtitles are positioned where the speaker is located
Subtitles with too much detail
Subtitles with too much detail
Subtitles with too much detail
Hard to tell movie from graphics!
Hard to tell movie from graphics!
Hard to tell movie from graphics!
Graphics that ruin the movie
Graphics that ruin the movie
Graphics that ruin the movie
Asking in the middle ruins immersion
Asking in the middle ruins immersion
Asking in the middle ruins immersion
Interactive AI search while watching
Interactive AI search while watching
Interactive AI search while watching
AirPods conflict with hearing aids, not usable
AirPods conflict with hearing aids, not usable
AirPods conflict with hearing aids, not usable
The vibration feels too simple and repetitive
The vibration feels too simple and repetitive
The vibration feels too simple and repetitive
Do
Do
Do
Keep subtitles at the bottom,
avoid variation
Keep subtitles at the bottom,
avoid variation
Keep subtitles at the bottom,
avoid variation
Describe only the current scene
Describe only the current scene
Describe only the current scene
Use only vibration as an extra
immersive cue
Use only vibration as an extra
immersive cue
Use only vibration as an extra
immersive cue
Review unclear parts after watching
Review unclear parts after watching
Review unclear parts after watching
Watch works best for haptic feedback
Watch works best for haptic feedback
Watch works best for haptic feedback
Detailed vibration feedback for each situation
Detailed vibration feedback for each situation
Detailed vibration feedback for each situation
User Flow
User Flow
STEP 1
STEP 1
Open a movie on Netflix
Open a movie on Netflix
STEP 2
STEP 2
Launch our app
Launch our app
STEP 3
STEP 3
Search for the movie title
Search for the movie title
Click!
Search
Click!
Conveying sound vibrations
to the users via a smartwatch.
Conveying sound vibrations
to the users via a smartwatch.
Conveying sound vibrations
to the users via a smartwatch.
Customizable subtitle settings adjust to user preference.
Customizable subtitle settings adjust to user preference.
Customizable subtitle settings adjust to user preference.
Genre-based templates
enable quick subtitle choice.
Genre-based templates
enable quick subtitle choice.
Genre-based templates
enable quick subtitle choice.
Musical
Musical
( 🎹 Piano melody, enjoyably playing)
( 🎹 Piano melody, enjoyably playing)
I will be back!
I will be back!
Action
Action
Run!!!
Run!!!
(Boom💣!)
(Boom💣!)
Drama
Drama
No..Don’t come here..
No..Don’t come here..
You are dead.
You are dead.
Romance
Romance
What are you drawing?
What are you drawing?
Bookmark scenes to
rewatch when sound is unclear.
Bookmark scenes to
rewatch when sound is unclear.
Bookmark scenes to
rewatch when sound is unclear.
User Feedback
User Feedback
User Feedback
*Permission to use interviewee photos in this portfolio has been obtained.
*Permission to use interviewee photos in this portfolio has been obtained.
Compared to existing OTT subtitles, the current version is better.
Compared to existing OTT subtitles, the current version is better.
Compared to existing OTT subtitles, the current version is better.
Strongly Disagree
Strongly Disagree
Strongly Disagree
1
1
1
2
2
2
Neutral
Neutral
Neutral
3
3
3
Agree
Agree
Agree
Disagree
Disagree
Disagree
4
4
4
5
5
5
Strongly Agree
Strongly Agree
Strongly Agree
The app makes it easier to understand a film’s emotions and context.
The app makes it easier to understand a film’s emotions and context.
The app makes it easier to understand a film’s emotions and context.
Strongly Disagree
Strongly Disagree
Strongly Disagree
1
1
1
2
2
2
Neutral
Neutral
Neutral
3
3
3
Agree
Agree
Agree
Disagree
Disagree
Disagree
4
4
4
5
5
5
Strongly Agree
Strongly Agree
Strongly Agree
The app does not disturb immersion when watching a movie.
The app does not disturb immersion when watching a movie.
The app does not disturb immersion when watching a movie.
Strongly Disagree
Strongly Disagree
Strongly Disagree
1
1
1
2
2
2
Neutral
Neutral
Neutral
3
3
3
Agree
Agree
Agree
Disagree
Disagree
Disagree
4
4
4
5
5
5
Strongly Agree
Strongly Agree
Strongly Agree
“Naturally got immersed as it conveyed emotions and sound effects missing in regular OTT subtitles.”
“Naturally got immersed as it conveyed emotions and sound effects missing in regular OTT subtitles.”
“Naturally got immersed as it conveyed emotions and sound effects missing in regular OTT subtitles.”
“Since it was customized, extra information appeared only when needed and didn’t break the flow.”
“Since it was customized, extra information appeared only when needed and didn’t break the flow.”
“Since it was customized, extra information appeared only when needed and didn’t break the flow.”
Impact
Impact
Impact
Satisfaction increased by 153%.
Satisfaction increased by 153%.
Satisfaction increased by 153%.
Q. How would you rate your satisfaction
with watching movies on OTT platforms?
Q. How would you rate your satisfaction with watching movies on OTT platforms?
Satisfaction score with
Regular OTT subtitles
Satisfaction score with
Regular OTT subtitles
1.9
1.9
8 respondents
8 respondents
2.5X
2.5X
Q. What is your satisfaction level with
the improved subtitles versus the original?
Q. What is your satisfaction level with the improved subtitles versus the original?
Satisfaction score with
Talebridge subtitles
Satisfaction score with
Talebridge subtitles
4.8
4.8
8 respondents
8 respondents
UI System
UI System
UI System
The design system was built from start to finish.
The design system was built from start to finish.
The design system was built from start to finish.
Accessibility
Accessibility
Accessibility
Integrated a11y checks to ensure accessible color contrast and readability.
Integrated a11y checks to ensure accessible color contrast and readability.
Integrated a11y checks to ensure accessible color contrast and readability.
What I’ve learned
What I’ve learned
What I’ve learned
Enhancing Immersion Without Distracting.
Enhancing Immersion Without Distracting.
Enhancing Immersion Without Distracting.
I realized that designing for accessibility means delivering the necessary information with clarity while preserving immersion without disrupting the flow of the movie. In particular, through the process of designing subtitles for hearing-impaired viewers, I learned that the effectiveness greatly depends on how visual and haptic elements are arranged to supplement emotions and context. Ultimately, accessibility design is about carefully balancing clarity and immersion, and I came to understand that achieving this balance is the true starting point for improving user experience.
I realized that designing for accessibility means delivering the necessary information with clarity while preserving immersion without disrupting the flow of the movie. In particular, through the process of designing subtitles for hearing-impaired viewers, I learned that the effectiveness greatly depends on how visual and haptic elements are arranged to supplement emotions and context. Ultimately, accessibility design is about carefully balancing clarity and immersion, and I came to understand that achieving this balance is the true starting point for improving user experience.
More projects
Hardware Interaction
A baby's cry causes analysis using a wearable device and an application
A baby's cry causes analysis using a wearable device and an application
A baby's cry causes analysis using a wearable device and an application
Helping understand babies better
Sponsored project, Hardware
Ideation Meeting Support Tool for ADHD
Ideation Meeting Support Tool for ADHD
Ideation Meeting Support Tool for ADHD
Helping Users 59.3% More Focused
Business Strategy
Built Personality-Level Trust in Dating Experiences
Built Personality-Level Trust in Dating Experiences
Built Personality-Level Trust in Dating Experiences
Achieved 40% increase in users
© Yunjin Tak 2025 All Rights Reserved
© Yunjin Tak 2025 All Rights Reserved