A Machine Learning Model called ‘Audeo’ made by the University of Washington recreates the atmosphere of that of a musical concert.
The bliss of a concert is always about the way the notes are played on the instruments which transforms mere finger movement into something divine for our ears. The people at the University of Washington thought about it too and said “Hey, why don’t we try to recreate that using a machine learning model?” And thus came about Audeo.
Audeo has been trained to learn music from a silent video of someone playing the piano and then replicating the piece played after learning. When tested with a music recognition app, it correctly identified the pieces that Audeo played about 86% of the time. Audeo was presented at the NeurIPS 2020 conference back in December.
“To create music that sounds like it could be played in a musical performance was previously believed to be impossible. An algorithm needs to figure out the cues, or ‘features,’ in the video frames that are related to generating music, and it needs to ‘imagine’ the sound that’s happening in between the video frames. It requires a system that is both precise and imaginative. The fact that we achieved music that sounded pretty good was a surprise.”
Senior author Eli Shlizerman, an assistant professor in both the applied mathematics and the electrical and computer engineering departments.
The steps that are used by Audeo to produce music is as follows. First, it will have to detect the keys that are being played per frame on the piano, so that it can create a diagram as the video progresses. Then it will have to translate that diagram into “synthesizer recognizable” music. Once that is done, it will have to clear the data and add in information – the tweaks that enhances music; such as how long a key is being pressed and how strong.
To train the model, they used Youtube videos of the pianist Paul Barton playing music of popular composers like Bach or Mozart. Two different synthesizers were used for the testing of the piece. Further research on more music transcribing of other pianists and instrumentalists is yet to be discovered.
“The goal of this study was to see if artificial intelligence could generate music that was played by a pianist in a video recording—though we were not aiming to replicate Paul Barton because he is such a virtuoso. We hope that our study enables novel ways to interact with music. For example, one future application is that Audeo can be extended to a virtual piano with a camera recording just a person’s hands. Also, by placing a camera on top of a real piano, Audeo could potentially assist in new ways of teaching students how to play.”
Eli Shlizerman; assistant professor in both the applied mathematics and the electrical and computer engineering departments
References:-
University of Washington.’ Audeo’ teaches artificial intelligence to play the piano. 04 Feb.2021: https://techxplore.com/news/2021-02-audeo-artificial-intelligence-piano.html