Robot Arm Control Platform

Multi‑modal control system for a robotic arm combining:

• EMG (muscle) signal acquisition via BITalino → feature extraction → RandomForest gesture inference
• Computer vision hand pose (MediaPipe + OpenCV) streaming per‑finger OPEN/CLOSED states
• Speech / STT control (Whisper API) mapping natural language commands to servo targets
• Manual / direct hand fallback for calibration & safety
• Microcontroller firmware (PlatformIO / main.cpp) executing a simple CSV angle protocol

Goal: unify heterogeneous human input channels into a reliable, low‑latency actuation pipeline with safety + extensibility.

Repository Layout

platformio.ini          # Firmware build config
src/
  main.cpp              # MCU firmware (serial CSV parser -> servos)
  emg_control.py        # EMG acquisition, filtering, feature, RF classify
  emg_plot.py           # Live oscilloscope for electrode placement
  hand_control.py       # MediaPipe hand pose -> finger angle states
  stt_control.py        # Speech-to-text -> gesture / angle mapping
  final_rf_model.pkl    # RandomForest model
  final_scaler.pkl      # Matching feature scaler

Control Modalities

1. EMG Gesture Control

1. Acquire raw EMG (BITalino) @ sampling rate
2. Notch (50/60 Hz) + bandpass (20–400 Hz) filtering
3. Window + extract time/frequency features
4. Scale (saved scaler) → RandomForest prediction
5. Majority vote + confidence threshold → gesture
6. Map gesture → 5 servo angles (e.g. 0,180,0,0,0)

Includes calibration tips, electrode placement guidance, and retraining workflow (collect → features → scaler+model pair).

2. Computer Vision Hand Control

MediaPipe Hands landmark inference → per‑finger OPEN/CLOSED heuristic → angle mapping (0°/180°) serialized to MCU. Future: proportional flexion via joint angle vectors + smoothing.

3. Speech / STT Control

Records short audio window → Whisper transcription → fuzzy intent match → angle sequence. Extensible via simple command map; can swap to offline engines (Vosk) for low‑latency / privacy.

4. Manual / Fallback & Safety

Direct angle streaming / emergency stop path retained; firmware watchdog returns to safe pose if no packets in window.

5. Firmware

Serial protocol: thumb,index,middle,ring,pinky\n (5 integers 0–180). Parses, clamps, actuates. Designed for modular servo mapping & future rate limiting.

Architecture

   +-------------------+        +------------------+
 EMG -->| emg_control.py     |--CSV-->|                  |
   +-------------------+        |                  |
 HandCV>| hand_control.py    |--CSV-->|  Microcontroller |--> Servos
   +-------------------+        |    (main.cpp)     |
 Speech>| stt_control.py     |--CSV-->|                  |
   +-------------------+        +------------------+

Key Features

• Pluggable multi‑input fusion (EMG / CV / STT)
• Consistent 5‑channel angle protocol
• Model + scaler pairing integrity safeguards
• Real‑time visualization (EMG oscilloscope)
• Extendable command & gesture mappings

Quick Start (Python Layer)

python -m venv .venv && .venv\\Scripts\\activate
pip install numpy scikit-learn matplotlib pyserial pybluez bitalino opencv-python mediapipe
python src/emg_plot.py   # verify signal quality
python src/emg_control.py
python src/hand_control.py
python src/stt_control.py

Future Enhancements

• Proportional flexion (continuous angles) via joint vector geometry
• Temporal smoothing and confidence decay logic
• Web telemetry dashboard + safety watchdog service
• Local STT / on‑device inference to cut latency

Safety Notes

• Always test new mappings at reduced torque
• Maintain emergency stop pathway & idle timeout

License & Repo

Source / updates: https://github.com/abduelmorsi/robot-arm-cv

Acknowledgments

BITalino toolkit, MediaPipe, OpenCV, scikit‑learn, Whisper API.

Outcomes

Improved positional accuracy & handling robustness in dynamic scenes; groundwork for future multi-object task orchestration.