Barrier-Free Video Accessibility

TEKNOFEST Finalist

WebSockets · Whisper API · Speech-to-Text · 3D Animation · Chrome Extension · TypeScript · Real-time Streaming · Accessibility · Kapalı Kaynak

Bu içerik yalnızca İngilizce olarak mevcuttur.

Barrier-Free Video Accessibility

TEKNOFEST Barrier-Free Living 2025 Finalist — A real-time accessibility platform converting video audio into sign language avatar animations for deaf & hard-of-hearing users.

Problem

Traditional captions are insufficient for many deaf users whose primary language is sign language. Existing solutions lack real-time avatar-based translation and broad platform compatibility.

Solution

Architected a real-time system that:

  • Translates spoken content into 3D Sign Language animations
  • Implements low-latency audio streaming using WebSockets
  • Integrates speech-to-text through the Whisper API
  • Provides seamless integration across web platforms through a Chrome Extension

Architecture

Architecture overview: browser audio flows through transcription and sign mapping to a 3D avatar overlay.

Architecture overview, not a product screenshot. View the TEKNOFEST participation certificate.

Chrome Extension
  ├─ Content script: audio capture / player hooks
  ├─ WebSocket client: receives animation frames
  └─ UI overlay: avatar canvas + controls

Backend Pipeline
  ├─ Audio chunk ingestion (stream endpoint)
  ├─ Whisper transcription
  ├─ Sign language mapping layer
  ├─ 3D avatar animation frame generator
  └─ Broadcast server (WebSocket)

Key Features

  • Low-latency audio → sign language pipeline
  • Whisper-powered multilingual transcription
  • Real-time 3D avatar animation rendering
  • Adaptive buffering & resilience strategies
  • Privacy-respecting (no raw storage of personal audio)

Impact

Expands accessibility of educational & informational video content while lowering cognitive load compared to reading-only captions.

Contribution and Validation

Implemented WebSocket audio streaming and speech-to-text integration for the browser extension and avatar pipeline. The project reached the TEKNOFEST 2025 Barrier-Free Living finals.

This is an accessibility prototype. The public materials do not report a sign-language comprehension study or a measured end-to-end latency distribution. User comprehension and sign-language quality need evaluation with sign-language users before making those outcome claims.