Interactive System （P5.js）
Brief
Create an interactive system (a game, a story, a performance, etc) that relies on the recognition and tracking of the real-world objects by a machine learning system (ML5js). Explore the possibilities of the technology: what it can and cannot do? how subtle can the variations be? what is easy and what is hard to do with the existing tech? Starting with your big idea outline a backup plan - what is plan B to scale it down? What is plan C? What will work if nothing else works?
Openprocessing link
https://openprocessing.org/sketch/1317973 ( Welcome to play it! )
Description
My inspiration comes from the piano. People can play in the air through human body recognition, so I create a touchable digital piano (a kind of game) that relies on the recognition and tracking of my wrists by ML5js. As the following video shows, I created seven circles of different colors to represent the seven scales of the piano. The left wrist can control the three circles C, D, and E on the left. The right wrist can control the four circles F, G, A, and B on the right.
My plan A is to create seven rectangles on the interactive interface to simulate the keyboard of the piano. Later, I found that many algorithms need to be considered when calculating the contact distance between the wrist and the rectangle, so I gave up the rectangle and adopted plan B. The circle only needs to consider the distance between the wrist and the radius of the center of the circle to make a sound.
As you can see from the video, although I used the circle as the piano keyboard, it is not very precise in the process of interaction. When touching the circles with different colors and scales, the orange ball on the wrist keeps touching the scales and the result is not very good. The user can interact with the interface to make the "piano" sound, but it does not play a tune smoothly.
Process
Set the piano scale in the sound library of P5.js
Set the human body to identify the wrist point for two small orange circles, the left hand to control the left three circles, the remaining right hand to control the right four circles.
When the orange circle of the wrist touches other circles, the circle will have a visual change in size.
Reflection
I think the most challenging part for me is to understand how to use the ml5 functions. For example, in my code, I used posenet to extract data from the video recorded. To start with I was very confused by this function and what parameters I needed to provide. Unlike traditional functions, this function is under the ML5 library which I needed to install first. After the installation, I googled online and eventually found a few good examples of how to use the function.