Gizmu
Interactive Music App in Augmented Reality
Team
Athena Hosek - Developer
Eva Chen - UX/UI Designer
James Lee - Developer
Kristine Yang - PM / Logo Designer
Marina Guthmann - VFX Designer
Yen-Yi Huang - UX/UI Designer
Tool
Unity
Figma
Adobe Illustrator
After Effect
Photoshop
Project Overview
Our project is about expressing yourself through the creation of music, with the help of everyday objects!
Anyone can be a composer and create interesting music with common objects that are accessible to anyone.
Check our demo video : Gizmu
Concept
Our project is about expressing yourself through music creation with the help of everyday objects!
Anyone can be a composer and create interesting music with common objects that are accessible to anyone.
Goal
Empower people to become composers.
Design Process
01
Discover
Explore design opportunities and potential constraints.
02
Define
Determine main features and user flow. Decide devices for better UX and feasibility for development.
03
Develop
Team developers
+ Team Figma
+ Team Unity
working in Agile work sprint.
04
Deliver
Minimum Viable Product
Discover
01
Magic Leap vs. Mobile Device
At first, Magic Leap was our chosen platform to enhance the immersive and intuitive experience by utilizing headset and hand interactions.
However, with the concern of limited time and the cons of Magic Leaps, we decided to switch to mobile devices due to developing familiarity and ease of user access.
Define
02
Features
As we had only 3 weeks, we made a list of the app features we required and prioritized three essential functions for our MVP:
Object Detection
Music Assignment
Sound Library
If time permits, we will continue to develop other advanced features related to personal accounts and recording.
User Flow
We organized the user flow based on the features listed, including the advanced account feature for future development. We designed the complete experience for the MVP, including onboarding and interacting with the app. The user flow would be:
Onboarding -> Detect Object -> Assign Sound -> Reassign Sound/ Play Music
Develop
03
Figma Prototype
Fig. Hi-Fi Prototype on Figma
Onboarding
Close-up view
↓
Object
Detection
Assign
Sound
Assign Sound
Play
Mode
Play Mode
Develop
03
Look & Feel
Logo & Naming
The name, GIZMU, is inspired by the lyrics "I've got gadgets and gizmos a-plenty" from The Little Mermaid's song "Part of Your World." Its name reflects its connection to both music and daily objects.
Color &
Typography
Design System to maintain consistency.
4 theme colors & gradient
Icons &
Components
Design System to maintain consistency.
4 Colors & Gradient
Other Elements
Design System to maintain consistency.
4 Colors & Gradient
Develop
Developing Process
Searched for Unity projects using YOLO obj detection
We tested YOLO, a cutting-edge object detection system that operates in real time. Utilizing a pre-trained model of a convolutional neural network, it can accurately identify objects from a list of pre-trained classes. However, during our testing, we encountered issues with freezing and incorrect object detection. As a result, we are exploring alternative methods for object detection.
Source: https://github.com/leggedrobotics/darknet_ros
2. Built basic functionality using placeholder objects
During this stage, we developed fundamental features such as positioning the menu on the designated objects and selecting songs from the sound library. We faced challenges such as ensuring the menu always remains oriented towards the camera and transitioning between various levels for different functions.
3. Got our selected object detector running
We have discovered a new method for detecting objects thanks to our skilled developer. This approach is much faster than YOLO. The item must be identified within the rectangular region in the center of the interface.
4. Combined object detection with basic functionality, setup control flow
Our developers have successfully combined object detection with basic functionality and established the control flow to connect these functions in this step.
Coding Chaos
Setting up and dealing with Unity and Git
Collaboration issues between Mac and Windows (solved with .gitignore)
Mysterious Unity/Xcode errors that fixed themselves
No debug log messages for Android
Limited error info for iOS
Waiting for builds
Some things in Unity worked exactly as expected, while others mysteriously didn’t work (for example, SetActive on a parent component vs. on each child)
Sleep-deprived developers make silly mistakes.
03
Deliver
Demo
This demo shows the following user flow:
Onboarding -> Detect objects -> Assign sounds -> Adjust Sounds (Volume, Play/Stop, reassign sound)
(PS. The demo is available on TestFlight for IOS users, and the APK file is provided for android users to test)
04
Reflection
Thanks team Gizmu!!!