Closed Source

Barrier-Free Video Accessibility Extension (TEKNOFEST 2025)

Express.js Whisper API OpenAI API Real-time Streaming Accessibility 3D Avatar Animation TypeScript Chrome Extension WebSockets

Barrier-Free Video Accessibility Extension

TEKNOFEST Barrier-Free Living 2025 finalist project. A Chrome extension & backend system that enhances video accessibility by converting spoken audio into real-time sign language avatar animations.

Problem

Traditional captions are insufficient for many deaf users whose primary language is sign language. Existing solutions lack real-time avatar-based translation and broad platform compatibility.

Solution

Pipeline converts streaming audio → text (Whisper) → sign instruction mapping → avatar animation commands delivered to extension front-end.

Architecture

Browser Extension
  ├─ Content script: audio capture / player hooks
  ├─ WebSocket client: receives animation frames
  └─ UI overlay: avatar canvas + controls
Backend (Express.js)
  ├─ Audio chunk ingestion (stream endpoint)
  ├─ Whisper transcription (OpenAI)
  ├─ Sign language mapping layer
  ├─ 3D avatar animation frame generator
  └─ Broadcast server (WS)

Features

  • Low-latency audio → sign pipeline
  • Whisper-powered multilingual transcription
  • Real-time 3D avatar animation
  • Adaptive buffering & resilience strategies
  • Privacy-respecting (no raw storage of personal audio)

Future Directions

  • Custom trained sign language model
  • Offline inference fallback
  • Multi-avatar personalization & styling
  • Integration with education platforms

Impact

Expands accessibility of educational & informational video content; lowers cognitive load vs reading-only captions.