Projet WAVE

ANR | ContInt 2013/2015

Visual Tools

Last updated by Victor Saiz on

Within the context of the WAVE project we have the need to represent and interact with different audio and musical data in the browser. For that reason during the past months we have been working on a small set of tools to help build simple time-based visualisations as well as fully fledged audio web applications.

For such collection of tools we can already point out the following requirements:

  • The tools should allow to represent and interact with several musical data-flows – self-hosted, streamed or hosted elsewhere – in an easy and reusable way.
  • The visualisations need to be easily incorporated in any web application and interact with other embedded multimedia content via event subscription or in some cases data bindings.
  • The visualisation should have the means to edit the data they represent

Thankfully many of the tools we are working on had been developed long before in-house for desktop environments, so in order to use their knowledge on audio-oriented graphical interfaces we discussed with the developers of audiosculpt and MuBu. On the web domain some sparse projects started tackling the same problematics, and while we follow and take all these in account, we believe our proposal is slightly different, in the way we designed a modular approach, emphasising on reusability, comparability and independence. We are building visual tools that anyone could use without having to load a monolithic framework or worse, be stuck with a particular application that tries to solve everything.

The technology

Because one of the key ideas of the WAVE project is to fully comply to web standards, the toolbox we will release will be written in ECMAScript 5 javascript and make use of all the modern browsers capabilities such as HTML5, CSS3 and other APIs.

After considering the options for web visualisations we found SVG to be the most convenient for most of our interfaces, since it integrates naturally with the browser’s event system and that just fits perfectly our interaction model. Down that path there are many SVG manipulation libraries we liked, however they just dealt with basic drawing and interaction at their core, and we knew we would face some classic data visualisation problematics such as data transformation, scales, axes, zoom et. al. For that reason we looked for specialised visualisation libraries. We chose to go with D3 since it offers an extensive toolbox for general purpose data visualisation, it’s well tested, it also has a very active community and every piece of functionality can be re-compiled by itself and included in a thinner component without having to include the entire library, which suites perfectly our packaging philosophy.

That said, we don’t discard using other graphical APIs like canvas/WebGL down the road for some of the components that won’t need heavy interaction/editing (this is something to take in consideration from today when working on the existing tools) and these APIs can offer better performance in some cases.

The Architecture

Since all of the components we envision so far happen in a time domain, the proposed structure relies on a main object that will keep the time axis consistent and will be responsible for common layout tasks such as zooming or the selection of components. Components will then be aded to this main object as layers via a plugin API. This API, once stable will allow developers to create their own components to extend the existing list.

wako.js

In the case you will want to use all the libraries together you will then include the packaged wako.js. This will be the namespace containing all the parts ready to use.

wako.timeLine

The wako.timeLine, a layout object, can incorporate any number of layers. Layers can have an independent vertical offset. This among other things lets you create complex timelines combining and overlapping such layers keeping the time synchronicity.

1
2
3
4
5

// let's create a timeline manager
var layout = wako.createTimeLine()
  .width(400)
  .height(100)
  .margin({top:10, left:10})

The wako.timeLine object can at the same time be included in other generic D3 or SVG visualisations as well since it complies with the already established D3 reusability proposal.

Layers are components

On the other hand, layers (components) will be in charge of their own display, edit and interaction capabilities.

1
2
3
4
5
6
7
8
9

// wako segment plugin
var segments = wako.createSegments({
  name: 'sections',
  data: data['sections']
});
// later in your timeline
var layout = wako.createTimeLine()
  .width(400)
  .layer(segments)

Components are also responsible for notifying their state via events to the external application.

The components

Here’s the current list of the components we will release:

Waveform

Simple waveform representation of (audio) data flows

Segments

Block-like segment editor/visualizer

Segmented audio

Combining behind the scenes segment and audio visualizers

(multi)Break-point

Visualisation and edition of one or multiple break-point functions

Markers

Simple marker editor/visualiser

Pianoroll

Simple midi notes visualizer/editor

Sonogram

Sonogram visualizer

User levels

Now when thinking on the potential users we could have besides ourselves we realised there are certain modules that could be useful in different scenarios outside our particular needs; even when we envisioned ready-to-use visualizers. So being able to export modules at different levels became very important. That brought us to target three different group of users:

Web Content editors

For the content editors we will package “finished” visualizers ready to use in multiple common formats such as wordpress plugins, substance nodes and other platforms. Our biggest bet however is on custom HTML elements “Custom Elements allow web developers to define new types of HTML elements”. We have already implemented some of our visualizers as custom elements using polymer and it’s undoubtedly the future, however some of the APIs are still unstable and we don’t want to fully rely on custom elements, plus it’s very easy to integrate any javascript code inside them. Anyways, by packaging plug-and-play components we will allow anyone to share and render their sound data very easily.

Application developers

For javascript developers we will be shipping the same javascript components with full javascript APIs so they can be included in your fully fledged application and easily scripted and hooked on to by simple javascript commands and DOM events.

Visualisation developers

Finally, on the lowest level, since our components are built on top of small bare bones D3 modules, visualisation developers wanting to tap into the lower level APIs will also have access to these small chunks of functionalities and use, compose and combine them as needed in their more complex visualisations.