Closed Source

Barrier-Free Video Accessibility (TEKNOFEST Finalist)

WebSockets Whisper API Speech-to-Text 3D Animation Chrome Extension TypeScript Real-time Streaming Accessibility

Barrier-Free Video Accessibility

TEKNOFEST Barrier-Free Living 2025 Finalist — A real-time accessibility platform converting video audio into sign language avatar animations for deaf & hard-of-hearing users.

Problem

Traditional captions are insufficient for many deaf users whose primary language is sign language. Existing solutions lack real-time avatar-based translation and broad platform compatibility.

Solution

Architected a real-time system that:

  • Translates spoken content into 3D Sign Language animations
  • Implements low-latency audio streaming using WebSockets
  • Delivers high-accuracy speech-to-text pipelines via Whisper API
  • Provides seamless integration across web platforms through a Chrome Extension

Architecture

Chrome Extension
  ├─ Content script: audio capture / player hooks
  ├─ WebSocket client: receives animation frames
  └─ UI overlay: avatar canvas + controls

Backend Pipeline
  ├─ Audio chunk ingestion (stream endpoint)
  ├─ Whisper transcription (high-accuracy STT)
  ├─ Sign language mapping layer
  ├─ 3D avatar animation frame generator
  └─ Broadcast server (WebSocket)

Key Features

  • Low-latency audio → sign language pipeline
  • Whisper-powered multilingual transcription
  • Real-time 3D avatar animation rendering
  • Adaptive buffering & resilience strategies
  • Privacy-respecting (no raw storage of personal audio)

Impact

Expands accessibility of educational & informational video content while lowering cognitive load compared to reading-only captions.