Human-Robot Collaboration to Design a Garden

This is a project (in the HRC^2 lab) I worked on for a few months before switching to my current (and more advanced) project. The idea is for a human and a robot to collaboratively design a small garden together without verbal communication. This required the robot to infer the human's design intent through the existing design and hand gestures by the human.

Accomplishments:
Built a multimodal AI framework that infers human intent from nonverbal cues by converting live video into structured text prompts for an LLM, enabling collaborative garden design with a Kinova Gen3 robotic arm. Engineered a real-time object and hand detection system (OpenCV + Google MediaPipe + ROS 2) sustaining 60 FPS. Cut perception-stack latency 120 ms → 40 ms via multithreaded pipelining and latest-frame policy (drops stale frames). Created a script that allowed the robotic arm to place 3D-printed fences based on the human's gestures. Delivered 7 ROS 2 nodes + 3 launch configurations, cutting demo setup time from ~8 min → 30 s (93% reduction).