Overview
OASIS is an interactive, maestro-like rhythm and gesture game developed during an intensive 1-week immersive residency. The project explores the intersection of spatial interaction and physical computing.
Using a standard camera, the game tracks the user’s hand movements in the real world to act as a conductor’s baton. By matching on-screen directional prompts, players build up a score that, when maxed out, triggers physical, real-world rewards via a custom-lit volcano or an automated bubble machine.
The Challenge
The core challenge was the extreme 1-week time constraint combined with a highly complex technical goal: building a flawless bridge between a digital software environment (Unity) and physical hardware components.
We had to translate erratic, raw video feed data into clean digital inputs, design an engaging gesture-matching gameplay loop, and engineer a reliable method for Unity to communicate with external hardware relays—all without the luxury of a long development and testing cycle.
Tech Stack
- Engine: Unity
- Language: C#
- Input/Tracking: MediaPipe
Features
- Camera-Based Gesture Tracking: Converts real-time hand movements into immediate directional game inputs without requiring physical controllers.
- Maestro Gameplay Loop: A responsive UI system featuring directional cues (arrows) and progressive scoring that reacts dynamically to player accuracy.
- Physical Hardware Integration: A digital-to-physical bridge that sends signals to ignite a custom-made volcano with LED lighting or activate a real-world bubble machine upon victory.
Technical Problems & Solutions
1. Translating Noisy Camera Feeds into Clean Game Inputs
- The Problem: Raw camera tracking data is inherently noisy and erratic. Small changes in room lighting or slight hand tremors caused the system to register rapid, accidental inputs, making the “maestro” conducting feel frustrating and unresponsive.
- The Solution: I implemented a custom input-filtering layer in C#. Instead of triggering an input the millisecond the hand crossed a boundary, I built a small data buffer that analyzed the velocity and consistency of the hand movement over a handful of frames, applying a smoothing algorithm to confirm an intentional “swipe.”
- The Result: We successfully filtered out background noise and accidental jitters, creating a stable and highly intentional gesture system that truly made the player feel like a maestro.
Lessons Learned
- Rapid Prototyping: This 1-week residency taught me how to scope a project. By prioritizing the core input/output loop first, we ensured we had a functional product early, leaving time to polish the visual experience.
- Designing for Immersive Spaces: I learned that player feedback shouldn’t just live on a monitor. Linking digital success to a physical bubble machine taught me a lot about spatial UX and how environmental feedback can vastly increase player delight.