This is my Master Thesis, submitted to National Tsing Hua University (Taiwan, July 2020). All the code can be found in this repository.

This thesis aims to use Machine Learning techniques to solve the novel problem of music interpolation composition. Two models based on Variational Autoencoders (VAEs) are proposed to generate a suitable polyphonic harmonic bridge between two given songs, smoothly changing the pitches and dynamics of the interpolation. The interpolations generated by the first model surpass a Random baseline data and a bidirectional LSTM approaches and its performance is comparable to the current state-of-the-art. The novel architecture of the second model outperforms the state-of-the-art interpolation approaches in terms of reconstruction loss by using an additional neural network for direct estimation of the interpolation encoded vector. Furthermore, the Hsinchu Interpolation MIDI Dataset was created, making both models proposed in this thesis more efficient than previous approaches in the literature in terms of computational and time requirements during training. Finally, a quantitative user study was done in order to ensure the validity of the results.

“Interpolation is a type of estimation, a method of constructing new data points within the range of a discrete set of known data points” (Fleetwood, 1991)

In traditional Machine Learning, the generation of music is conditioned on the past events. But what if we could condition the music generation on both past and future events? We would input a begin track and an end track of 10 seconds each to our model, obtaining the middle (or interpolation) track of also 10 seconds as output, whose pitches and dynamics match both given tracks.

In monophonic music, every timestep or time unit contains one single note. On the other hand, in polyphonic music, every timestep contains several notes, forming chords that make the composition richer. We use MIDI (Musical Instrument Digital Interface) to represent the music in a symbolic way, instead of using the raw waveform (which is computationally expensive to manipulate). Each timestep of a song is represented as a vector of 64 binary elements, where each binary element represents one piano key (or one note or pitch), 1 meaning note on and 0 meaning note off.

A new MIDI dataset based on the Lahk MIDI Dataset has been created: the Hsinchu Interpolation MIDI Dataset. This dataset contains only very valuable interpolation segments, where the begin track and the end track are very different (simulating a style transfer within the same human composition). The begin and end tracks similarity has been evaluated with a neural network (binary classification problem). The Hsinchu Interpolation MIDI Dataset contains 30,830 segments of 30 seconds each.

Four experiments have been done in this thesis. Experiments 1 and 2 are the baselines. Experiments 03 and 04 are proposed novel models based on Variational Autoencoders to solve the interpolation problem:

1. Random Data
2. Bi-LSTM
3. VAE (Variational Autoencoder)
4. VAE+NN (Variational Autoencoder + Neural Network)

1. Encode begin track and end track with VAE to obtain z_begin and z_end, respectively:

2. Average vectors z_begin and z_end to obtain the interpolation encoded vector z_interpolation:

3. Decode the interpolation encoded vector z_interpolation to obtain the interpolation track:

4. Ideally, the reconstructed interpolation track has to be identical to the original interpolation track (ground truth):

1. Encode begin track and end track with VAE to obtain z_begin and z_end, respectively:

2. Use the novel neural network (NN) approach to directly estimate the interpolation encoded vector z_interpolation based on z_begin and z_end:

3. Decode the interpolation encoded vector z_interpolation to obtain the interpolation track:

4. Ideally, the reconstructed interpolation track has to be identical to the original interpolation track (ground truth):

Full architecture of the novel VAE+NN model proposed in this thesis:

1. Averaged MSE error of each model's tracks:

2. MSE error by binarization threshold of each model's tracks:

3. MSE error of latent space of VAE models:

1. Pair-wise comparisons of each model's tracks:

2. Total number of votes per model: