Signal reconstruction from spectrograms

2018-05-31 13:53

Carl Thomé

Signal reconstruction from spectrograms¶

Reconstruct waveform from input spectrogram by iteratively minimizing a cost function between the spectrogram and white noise transformed into the exact same time-frequency domain.

Assuming 50% magnitude overlap and linearly spaced frequencies this reconstruction method is pretty much lossless in terms of audio quality, which is nice in those cases where phase information cannot be recovered.

Given a filtered spectrogram such as with a Mel filterbank, the resulting audio is noticeably degraded (particularly due to lost treble) but still decent.

The biggest downside with this method is that the iterative procedure is very slow (running on a GPU is a good idea for any audio tracks longer than 20 seconds) compared to just having an inverse transform at hand.

Reference¶

Decorsière, Rémi, et al. "Inversion of auditory spectrograms, traditional spectrograms, and other envelope representations." IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP) 23.1 (2015): 46-56.

In [1]:

from IPython.display import Image
Image('diagram.png')

Out[1]:

No description has been provided for this image

In [2]:

import tensorflow as tf


def sonify(spectrogram, samples, transform_op_fn, logscaled=True):
    graph = tf.Graph()
    with graph.as_default():

        noise = tf.Variable(tf.random_normal([samples], stddev=1e-6))

        x = transform_op_fn(noise)
        y = spectrogram

        if logscaled:
            x = tf.expm1(x)
            y = tf.expm1(y)

        x = tf.nn.l2_normalize(x)
        y = tf.nn.l2_normalize(y)
        tf.losses.mean_squared_error(x, y)

        optimizer = tf.contrib.opt.ScipyOptimizerInterface(
            loss=tf.losses.get_total_loss(),
            var_list=[noise],
            tol=1e-16,
            method='L-BFGS-B',
            options={
                'maxiter': 1000,
                'disp': True
            })

    with tf.Session(graph=graph) as session:
        session.run(tf.global_variables_initializer())
        optimizer.minimize(session)
        waveform = session.run(noise)

    return waveform

In [3]:

import librosa as lr

sample_rate = 44100
path = lr.util.example_audio_file()
waveform = lr.load(path, duration=3.0, sr=sample_rate)[0]


def logmel(waveform):
    z = tf.contrib.signal.stft(waveform, 2048, 1024)
    magnitudes = tf.abs(z)
    filterbank = tf.contrib.signal.linear_to_mel_weight_matrix(
        num_mel_bins=80,
        num_spectrogram_bins=magnitudes.shape[-1].value,
        sample_rate=sample_rate,
        lower_edge_hertz=0.0,
        upper_edge_hertz=8000.0)
    melspectrogram = tf.tensordot(magnitudes, filterbank, 1)
    return tf.log1p(melspectrogram)


with tf.Session():
    spectrogram = logmel(waveform).eval()

reconstructed_waveform = sonify(spectrogram, len(waveform), logmel)

INFO:tensorflow:Optimization terminated with:
  Message: b'STOP: TOTAL NO. of ITERATIONS EXCEEDS LIMIT'
  Objective function value: 0.000000
  Number of iterations: 1001
  Number of functions evaluations: 1059

In [4]:

from IPython.display import display, Audio

display(Audio(waveform, rate=sample_rate))
display(Audio(reconstructed_waveform, rate=sample_rate))

Signal reconstruction from spectrograms¶

Reference¶

Comments