JS-Xtract and Web Audio

The Web Audio API enables true, high performance rendering inside the web browser. However, there is no feature extraction possible inside the API.

JS-Xtract attaches prototypes to two of the native nodes: AudioBuffer and Analyser. These two nodes enable realtime and offline processing.

AudioBuffer

The AudioBuffer is an interface to hold PCM data for playback by a BufferSourceNode. The PCM data in the node can be extracted and processed using two extra fuctions.

Getting the data

The data can be extracted into a TimeData object for processing using AudioBuffer().xtract_get_data_frames(frame_size, hop_size). This function decimates the array in time and returns an array of TimeData objects holding the audio. The frame_size must be specified. If the hop_size is undefined, then hop_size = frame_size.

For example:

// Create a buffer of 1 second long. Let's assume buffer already has data!
var buffer = AudioContext().createBuffer(44100,1,44100);

var frames = buffer.xtract_get_data_frames(4096);

At the end, frames holds an array of 11 TimeData() objects, each holding 4096 samples of data. Each of these can be processed as needed, following the TimeData() object guide (link).

An alternative to this is to decimate the buffer directly. This is useful to use the procedural call style.

// Create a buffer of 1 second long. Let's assume buffer already has data!
var buffer = AudioContext().createBuffer(44100,1,44100);

// Copy the PCM data in channel 0 into the variable data
var data = buffer.getChannelData(0);

var frames = data.xtract_get_data_frames(4096);

In this case, frames holds an array of Float32Array objects, each holding 4096 elements. These can then be processed using the procedural function calls, or other objects.

Processing the data

The AudioBuffer also has a synchronous callback function appended as well, AudioBuffer().xtract_process_frame_data(func, frame_size, hop_size, arg_this). The first argument is the function to call. Then the frame and hop sizes. The arg_this allows you to specify a specific this object to use rather than the AudioBuffer instance.

The function callback is passed the following arguments:

An example function call to get the mean and variance would be:

function(element, channel, index, array) {
    element.mean();
    element.variance();
}

Actually, the element.mean() would not be needed since the variance will compute this, see here. Equally, the header could be function(element) since the other three arguments are not processed.

Because the function call is synchronous, the returned value comes from the AudioBuffer instance on which this was invoked. The returned value is an object of the following structure:

{
    "num_channels": 1,
    "channel_results": [{
        "num_frames": 1,
        "results": [{
            "mean": 0.4567,
            "variance": 0.1234
        }]
    }]
}

The returned results are the results nodes of each TimeData element.

Analyser

The Web Audio Analyser provides a way for extracting the current audio frame as either time or frequency domain arrays. However there is no built-in callback function to automate collection, instead being done 'on-demand'. The JS-Xtract library provides an asynchronous callback method here based on a high accuracy timer. The timer is actually a ScriptProcessNode triggering the supplied callback each time enough samples has iterated.

The callback is Analyser().frameCallback(func, arg_this). The func argument takes a function to call, arg_this the desired this statement.

The passed function is called with the following arguments:

The frequency domain information is already processed and stored in TimeData().result.spectrum as if calling TimeData().spectrum() has already happened. Likewise, calling TimeData().spectrum() will not do anything (since spectrum is already defined).

An example for setting up this node is as follows:

var analyser = audioContext.createAnalyserNode();

analyser.frameCallback(function(current, previous, previous_result){
    return current.variance();
});

// On next iteration the previous_result == the variance

It is possible to return an object, array or other complex types. However, if a number is passed back, then the analyser node can forward this information into the audio graph. For instance, the RMS amplitude in decibels can be extracted as:

var analyser = audioContext.createAnalyserNode();

// Bind it into your audiograph
some_audio_node.connect(analyser);

// Build the callback
analyser.frameCallback(function(current){
    return Math.pow(10.0, current.rms_amplitude() / 20.0 );
})
// Returns the dB value

// Now connect the hidden script-node into the next node
analyser.callbackObject.connect(next_audio_node);

Performing Deltas

Both the AudioBuffer and AnalyserNode support calculating delta's with ease, however it is not always clearly obvious. Using the Offline AudioBuffer as an example:

var buffer = AudioBuffer();

var results = buffer.xtract_process_frame_data(function(element, channel, index, array){
    element.mean();
}, 4096);

// Because of the synchronous callback, the results node holds all the means for the decimated buffer
// Therefore we can iterate
var prev_frame;
for (var c=0; c<results.num_channels; c++) {
    var chan = results.channel_results[c];
    for (var n=0; n<chan.num_frames; n++) {
        var frame = chan[n];
        frame.deltas = {};
        if (prev_frame) {
            frame.deltas.mean = frame.mean - prev_frame.mean;
        }
        prev_frame = frame;
    }
}

This will give an output, for instance as follows. If results were:

{
    "num_channels": 1,
    "channel_results": [{
        "num_frames": 1,
        "results": [{
            "mean": 0.1,
            "variance": 0.0
        },
        {
            "mean": 0.2,
            "variance": 0.5
        }]
    }]
}

, then the new results after delta would be:

{
    "num_channels": 1,
    "channel_results": [{
        "num_frames": 1,
        "results": [{
            "mean": 0.1,
            "variance": 0.0
            "deltas": {}
        },
        {
            "mean": 0.2,
            "variance": 0.5
            "deltas": {
                "mean": 0.1,
                "variance": 0.5
            }
        }]
    }]
}

Of course the code will need to behave differently for array deltas but the premise is the same.

For the realtime version, the code will have to behave as follows:

var analyser = audioContext.createAnalyser();

analyser.frameCallback(function(current_frame, previous_frame, previous_result){
    var result_object = {};
    result_object.mean = current_frame.mean();
    if (typeof previous_result == "object" && previous_result.mean) {
        result_object.delta = result_object.mean - previous_result.mean;
    }
    return result_object;
})

This will return an object, which will be passed on to the next function. Obviously, since it is asynchronous, to use this information you would want to bind it in to something. The following example will create an analyser node, compute the rms amplitude delta and control a gain node.

var analyser = audioContext.createAnalyser();
var gainNode = audioContext.createGain();

function callback(current_frame, previous_frame, previous_result) {
    current_frame.rms_amplitude();
    // current_frame.result.rms_amplitude is populated
    // we passed the previous frame, so it also has result.rms_amplitude
    // but in case it doesn't, we must sanity check
    var delta = 0;
    if (previous_frame) {
        // If we are here, then this must have run at least once
        delta = current_frame.result.rms_amplitude - previous_frame.result.rms_amplitude;
    }
    // If here, it hasn't run, so delta is 0.
    // The delta is all linear, so as long as the rms_amplitude is between [0, 1] then the gain will be the inverse of this
    return 1-delta;
}

analyser.frameCallback(callback);

// Connect the output to the gain node gain AudioParameter
analyser.callbackObject.connect(gainNode.gain);

This shows also how the object-oriented approach automatically collects the data for us. The following example does the same thing, except it passes it through an object using the this options:

var analyser = audioContext.createAnalyser();
var my_gain = {
    gain: audioContext.createGain(),
    callback: function (current_frame, previous_frame, previous_result) {
        current_frame.rms_amplitude();
        // current_frame.result.rms_amplitude is populated
        // we passed the previous frame, so it also has result.rms_amplitude
        // but in case it doesn't, we must sanity check
        var delta = 0;
        if (previous_frame) {
            // If we are here, then this must have run at least once
            delta = current_frame.result.rms_amplitude - previous_frame.result.rms_amplitude;
        }
        // If here, it hasn't run, so delta is 0.
        // The delta is all linear, so as long as the rms_amplitude is between [0, 1] then the gain will be the inverse of this

        // Note we are using this
        this.gain.gain.value = delta;
    }
}

analyser.frameCallback(callback, my_gain);

In this example the delta is being applied into the gain node by the callback function itself, this is because the 'this' parameter was set to the object my_gain.

From this it should be trivial to see how delta-deltas can be computed as it is the same loop but one more step.