During the pandemic in 2022, most people who lived in Shanghai were quarantined at home and it’s hard for individuals to engage in any art experience. By contrast, people can freely go to art galleries or travel to other countries before covid19. So I think it would be a good idea to create a virtual art & travel experience that people can consume via digital media without any physical limitation.
Client: Personal peoject
Role: 3D Design , Unity Programing , UI/UX
Watch the game play video above or on youtube
I need to show a nature environment that can help people to relax. Occasionally, I found this otherworldly landscape of Brazil's Lençóis Maranhenses National Park. When it comes to art experience, the first thing that came to my mind is about James Turrell and his lighting installation. I had a chance to visit his exhibition several years ago and the immersive lighting experience really blew my mind. Lastly, I also want to make some interactive music experience, so I thought it would be interesting to turn an old vinyl recorder to an interactive music installation.
To make a more meaningful and rich experience, I created a three-chapter story that represents different stages of life and encourages people to reflect on their lives.
1. Exploration
An installation made of a grid of arrows are floating in the air. All the arrows stays in the same direction like an army. But when the player hit the button, all arrows start to move in different direction, as if seeking for their unique direction.
2. Gestation
After passing through a long trestle, the player will get into a hall where noting is displayed except the light. This helps the player to focus more on his/her sensation and inner feelings just like meditation. The shape of the hall is like a womb, which makes people feel warm and safe and the interior light will changes rhythmically.
3. Decision
When arrived at the roof of the hall, the player can place stones on the record. Once the stone go through the colored gate, the record will make a sound based on the stones position on the disc. There are two instruments available to help player make a more musical sound. I also limited the maximum notes number because you can’s have endless choice in your life.
I started by exploring the 3d formations of the main building because it's the main part of the experience. I’d like to use a more organic shape to to represent the shape of a womb which always make people feel warm and safe. Inspired by the famous lighting artist James Turrell, I think the stairs and skylights perfectly match the womb concept. After the ideation part, I created the 3d building model in Rhino.
For the dune landscape, I use the unity built-in terrain tools, which helps me easily create a large environment with good level of details and flexibility. For the water shader, I use some free assets from the unity assets store.
Then, I created a vinyl like roof which is the main object for the next chapter. Instead of creating a typical recording head for a traditional vinyl player, I decided to add a door on the roof as a symbol for the gates towards future. Once the stone goes through the gate, the sound will be played.
key features:
- a third person controller to control the main character.
- a random movement system for the arrows in the first chapter triggered by a button click event
- a dynamic lighting system which changes its color along with time repetitively.
- a system to hold rigid body on a platform (unity’s default physics system doesn’t support this, when you rotate the platform, the object above will not move with the platform properly)
- a music creation system that allow user to place objects on the vinyl disc as music notes and play automatically.
The most interesting part is the idea of creating music by place notes on a round vinyl record. The note stone will be spawned on the disc according to the current mouse position. I use unity’s raycast to detect the 3d position of the hit point and add a tag to the disc so that it will prevent notes be spawned outside the disc.
In order to trigger different pitch of notes, the distance from the individual note to the center of the disc was calculated and remap to 12 piano notes in c major scale.
A collision was triggered once the the stone note hit the colored door and a note sound will be played at the same time.
I also limited the number maximum notes on the disc. Each time a stone was spawned, the number will decrease by one and this number will then be displayed on the UI.
Below is the script for the interactive music creation system.
The main user interface was placed in 3d world coordinate so it can be better blended with the 3d environment and provide a more immersive experience. In the future, I’d also like to make a VR version for this game, so a 3d UI is the best choice.
A 2d screen UI was created to help user to switch between stones types.
The game experience is incomplete without soundscape. I created 3 different background music for each chapters and use a box collider as trigger to automatically switch the music between different scenes. For the second chapter, I compose some simple chord progression with a mysterious electronic sound to help player to focus and listen to their inner voice.
Two music instrument were used in the third chapter to help player make a more rich sound. In order to make it more easily to produce musical compositions, only C major scale notes were used. The pad notes were one octave lower than the piano notes. For the piano, I chose a taped piano which is well suited for this vinyl style as a lead sound. All the notes were then recorded separately in Logic Pro.
In the early stage of the concept, I also came up with other ideas about how to express oneself by using music. In this attempt, people can play with a real midi keyboard and the signal will be transferred to unity and then use to trigger the ball. The size of the ball can change according to the velocity of the note. I also made some variations of the material color for each spawned ball.
I also made a virtual keyboard in the scene. When people press the key in the physical environment, the virtual keyboard can act responsively. This was achieved by reading midi signal from the keyboard and then using the signal to change each key in the virtual keyboard.
This piece of code was created to call the function of playing audio clips, changing key status and spawning balls when received a midi event. Most of the function was in another separate NotePlayer.cs class in order to reduce dependency and made the code more modular.
This is the NotePlayer.cs class which contains the functions for manipulating the keys and playing sound.
Here is the video demonstration for this interactive music experience.