Evaluation

Finally, we evaluate the results of sound matching from both the deep learning and genetic algorithms tested.

The MFCCEval class is used to perform an objective evaluation of each audio file generated. This is carried out by measuring error metrics between the MFCCs from the target sounds and the results from sound matching.

The results of this evaluation are saved in a JSON file which summarizes the results for each estimator.

Histogram plots are also used to show the distribution of results produced by each estimator. Histogram plots show the mean absolute error of the sound matched results from each estimator.

import spiegelib as spgl
import numpy as np
import matplotlib.pyplot as plt

# Load the sound targets used for sound matching
targets = spgl.AudioBuffer.load_folder('./evaluation/audio')

# Load all the estimations of the sound targets made by each estimator
estimations = [spgl.AudioBuffer.load_folder('./evaluation/mlp'),
               spgl.AudioBuffer.load_folder('./evaluation/lstm'),
               spgl.AudioBuffer.load_folder('./evaluation/bi_lstm'),
               spgl.AudioBuffer.load_folder('./evaluation/cnn'),
               spgl.AudioBuffer.load_folder('./evaluation/ga'),
               spgl.AudioBuffer.load_folder('./evaluation/nsga')]

# Evaluate the results and save to JSON file
evaluation = spgl.evaluation.MFCCEval(targets, estimations)
evaluation.evaluate()
evaluation.save_stats_json('./evaluation/evaluation_stats.json')

MLP Histogram

bins = np.arange(0, 40, 2.5)
evaluation.plot_hist([0], 'mean_abs_error', bins)

LSTM Histogram

evaluation.plot_hist([1], 'mean_abs_error', bins)

LSTM++ Histogram

evaluation.plot_hist([2], 'mean_abs_error', bins)

CNN Histogram

evaluation.plot_hist([3], 'mean_abs_error', bins)

GA Histogram

evaluation.plot_hist([4], 'mean_abs_error', bins)

NSGA III Histogram

evaluation.plot_hist([5], 'mean_abs_error', bins)