Edge-first smart home. Voice, gesture, and gas alerts — no cloud required.
24 weeks
9
In Progress
Key Challenge
Normalizing heterogeneous AI outputs to canonical MQTT tokens with hold-frame debouncing and FSM-safe routing
Skills Demonstrated
Building an edge-first smart home where voice and gesture inputs normalize to MQTT tokens, route through a Guardian FSM dispatcher, and drive NodeMCU actuators (door, fan, gas sensor) with alert escalation and E2E test coverage.
Cloud-dependent smart home systems fail offline and add latency to safety-critical actuator commands
Edge-first pipeline with offline Vosk/MediaPipe and local MQTT dispatcher FSM
Sub-second voice-to-actuator routing with automatic gas alert escalation
<1s
Voice-to-MQTT latency
6
Guardian FSM modes
<1s
Voice-to-actuator latency
30–50ms
Gesture inference per frame
20–30 FPS
Camera throughput on Pi 4
30s
Alert→Emergency countdown
6
FSM modes
13
MQTT topic categories
Operator dashboard with device states
Working edge smart home pipeline with voice, gesture, gas sensor, and actuator control.
Demonstrated offline-capable home automation without cloud dependency.
<1s
Command latency
3
Input modalities
Edge-first architecture with Raspberry Pi running voice/gesture publishers and the dispatcher ActionRouter.
All inputs normalize to canonical tokens before hitting MQTT. The Guardian FSM manages mode transitions with Alert→Emergency escalation after 30s countdown.
NodeMCU firmware modules subscribe to actuator topics and publish state feedback.
Infrastructure & Deployment
Raspberry Pi 4 on local network with Mosquitto MQTT broker. NodeMCU devices on WiFi. Flask dashboard on Pi. Node-RED for demo automation flows.
Vosk STT with expanded command vocabulary normalized to tokens.
MediaPipe Hands with hold-frame debouncing and cooldown.
Six modes with Alert→Emergency escalation and cancel support.
Modular firmware for door relay, fan relay, and gas sensor.
Flask web UI for live device states and manual overrides.
Remote control and notifications via Telegram.
Motion detection integrated into FSM triggers.
DHT11/22 readings published to MQTT.
The Problem
Natural speech has infinite phrasings for the same intent (turn on the fan vs fan on please).
How I Solved It
Expanded canonical vocabulary with fuzzy matching and phrase normalization before MQTT publish.
def normalize_command(text: str) -> str | None:
text = text.lower().strip()
for phrase, token in VOCABULARY.items():
if phrase in text:
return token
return NoneThe Problem
Hand jitter and partial gestures triggered unintended commands.
How I Solved It
Require GESTURE_HOLD_FRAMES (8) stable frames plus GESTURE_COOLDOWN_S (2.0) after each send.
if stable_frames >= HOLD_FRAMES and time.time() - last_sent > COOLDOWN:
publish_token(token)
last_sent = time.time()The Problem
Gas alerts need automatic response but users must be able to cancel false alarms.
How I Solved It
FSM enters ALERT, triggers fan_on, starts 30s countdown to EMERGENCY; alert_cancel token resets state.
def on_gas_detected(self):
self.transition('ALERT')
self.publish('fan_on')
self.start_countdown(30, target='EMERGENCY')The Problem
Dispatcher didn't know if actuator commands actually executed.
How I Solved It
NodeMCU publishes state feedback topics after relay actuation; dispatcher subscribes for confirmation.
Token normalization is essential
Voice phrases and gesture classifications must map to the same token set so the dispatcher treats all inputs identically.
Hold frames prevent accidental triggers
Requiring 8 stable gesture frames and 2s cooldown eliminated false positives from hand jitter.
FSM modes need explicit escalation
Alert mode with countdown-to-Emergency gives users time to cancel while ensuring safety fallback.
E2E tests on MQTT topics
Testing at the topic level catches routing bugs that unit tests miss.
What I'd Do Differently
Train a small gesture classifier instead of rule-based landmarks, and add motion/temperature sensors to complete the Guardian FSM sensor triggers.
A large Express.js + MongoDB backend that manages university academic structure, teacher seance scheduling with automatic conflict detection, payroll-grade financial aggregation, and an experimental genetic-algorithm timetable generator.
Healthcare pharmacy platform with patient/pharmacy accounts, prescriptions, orders, and product catalog.
Interested in a project like this or want to collaborate on your own?