Why Mimi?

Imagine this: you stand, center stage, in an opera house filled with admiring fans. Your voice soars over the the shimmering textures of the strings and intertwines with solo lines in the winds, borne up by the quiet strength of the brass. The orchestra is led by a sensitive conductor who follows, even anticipates, your most subtle nuance and gesture.

The goal of my current research project is to package this fantasy and deliver it to your doorstep in a box. Everything except the fans, that is. The box requires Linux, of course.

In this musical example I have taken the beloved aria "Mi Chiamano Mimi" from Puccini's La Boheme. I begin with an audio recording of the orchestral accompaniment, courtesy of the folks at Music Minus One and the acclaimed Festival Symphony Orchestra of Bulgaria. I then use a score matching algorithm to find the alignment between the score and the audio file. A technique known as "phase vocoding" allows me to play the audio file back at variable rate without introducing pitch distortions.

The main challenge is controlling the time varying audio play rate throughout the evolution of the performance. A mathematical model combines the real-time analysis of the soloist's performance with examples of past performances to produce a sequence of local goals for the accompaniment, much like a trail of bread crumbs. The phase vocoder adjusts the play rate to follow this trail.

Here is the performance. Hope you like it.
"Mi Chiamano Mimi" from Puccini's La Boheme

More musical examples of my accompaniment research

More involved discussions of the science behind this work