Memory Palace: YOLO Object Detection to Locate Objects through Speech Recognition

Winning submisson at Cal Hacks 5.0 (Epson sponsor award). Our team developed a pair of display goggles that helps patients with Alzheimer’s or other visual/mental disabilities in locating their personal belongings. Users verbalize the item they are searching for (typically items from the COCO image dataset), and all objects detected would be highlighted.

code installation

Technical summary

  • Trained Mask R-CNN object detection classifier on grayed-out ImageNet dataset to sustain realtime inference / classification rate at least 30fps; optimized setup of models (e.g. Yolo v1-3), datasets (e.g. MS Coco) on high-latency Android embedded system

  • Built Android application and custom scripts (for parsing and relaying camera input) and root-installed into Moverio augmented reality display googles to stream camera input, apply bounding boxes around objects to be detected, and output to display feed of device

  • Contributed an optimized low-latency embedded-system implementation that performs speech recognition, video streaming/display, image processing with minimal visible stutter


This is what the user sees.

Over 24 hours our team built the embedded system that would permit Epson’s Moverio goggles to detect specific objects for users. This included setting up voice recognition input for object commands, relaying the image from the goggles to a server PC, running an object detection system on the PC, and relaying the image in real-time back to the goggles. The object detection system was based on a Mask RCNN model trained on a modified image dataset.

Completion of the object detection system backend:

Preview of my teammate testing the goggles out in real-time, while my PC displays what he is seeing (Click the video to watch).


To install the object detection and streaming system for other AR-capable devices, you may (1) download the source code and (2) follow the below installation instructions.

CalHacks submissionCalHacks slidesRecount of Cal Hacksspinoff slides