Within the context of the WAVE project we have the need to represent and interact with different audio and musical data in the browser. For that reason during the past months we have been working on a small set of tools to help build simple time-based visualisations as well as fully fledged audio web applications.
For such collection of tools we can already point out the following requirements:
- The tools should allow to represent and interact with several musical data-flows – self-hosted, streamed or hosted elsewhere – in an easy and reusable way.
- The visualisations need to be easily incorporated in any web application and interact with other embedded multimedia content via event subscription or in some cases data bindings.
- The visualisation should have the means to edit the data they represent
Thankfully many of the tools we are working on had been developed long before in-house for desktop environments, so in order to use their knowledge on audio-oriented graphical interfaces we discussed with the developers of audiosculpt and MuBu. On the web domain some sparse projects started tackling the same problematics, and while we follow and take all these in account, we believe our proposal is slightly different, in the way we designed a modular approach, emphasising on reusability, comparability and independence. We are building visual tools that anyone could use without having to load a monolithic framework or worse, be stuck with a particular application that tries to solve everything.
After considering the options for web visualisations we found SVG to be the most convenient for most of our interfaces, since it integrates naturally with the browser’s event system and that just fits perfectly our interaction model. Down that path there are many SVG manipulation libraries we liked, however they just dealt with basic drawing and interaction at their core, and we knew we would face some classic data visualisation problematics such as data transformation, scales, axes, zoom et. al. For that reason we looked for specialised visualisation libraries. We chose to go with D3 since it offers an extensive toolbox for general purpose data visualisation, it’s well tested, it also has a very active community and every piece of functionality can be re-compiled by itself and included in a thinner component without having to include the entire library, which suites perfectly our packaging philosophy.
That said, we don’t discard using other graphical APIs like canvas/WebGL down the road for some of the components that won’t need heavy interaction/editing (this is something to take in consideration from today when working on the existing tools) and these APIs can offer better performance in some cases.
Since all of the components we envision so far happen in a time domain, the proposed structure relies on a main object that will keep the time axis consistent and will be responsible for common layout tasks such as zooming or the selection of components. Components will then be aded to this main object as layers via a plugin API. This API, once stable will allow developers to create their own components to extend the existing list.
In the case you will want to use all the libraries together you will then include the packaged
wako.js. This will be the namespace containing all the parts ready to use.
The wako.timeLine, a layout object, can incorporate any number of layers. Layers can have an independent vertical offset. This among other things lets you create complex timelines combining and overlapping such layers keeping the time synchronicity.
1 2 3 4 5
The wako.timeLine object can at the same time be included in other generic D3 or SVG visualisations as well since it complies with the already established D3 reusability proposal.
Layers are components
On the other hand, layers (components) will be in charge of their own display, edit and interaction capabilities.
1 2 3 4 5 6 7 8 9
Components are also responsible for notifying their state via events to the external application.
Here’s the current list of the components we will release:
Simple waveform representation of (audio) data flows
Block-like segment editor/visualizer
Combining behind the scenes segment and audio visualizers
Visualisation and edition of one or multiple break-point functions
Simple marker editor/visualiser
Simple midi notes visualizer/editor
Now when thinking on the potential users we could have besides ourselves we realised there are certain modules that could be useful in different scenarios outside our particular needs; even when we envisioned ready-to-use visualizers. So being able to export modules at different levels became very important. That brought us to target three different group of users:
Web Content editors
Finally, on the lowest level, since our components are built on top of small bare bones D3 modules, visualisation developers wanting to tap into the lower level APIs will also have access to these small chunks of functionalities and use, compose and combine them as needed in their more complex visualisations.