-
Notifications
You must be signed in to change notification settings - Fork 2
Home
The project aims to upgrade the former JavaScript wrapper of libfaust, WebAssembly version of Faust compiler generated by Emscripten and LLVM. libfaust backend transforms Faust codes to equivalent WebAssembly modules which are capable to process signals by passing buffers. These modules should be implemented later in WebAudio nodes in order to process audio signals in real-time within a local web environment. Modern tool-chain is used here to ensure the efficiency and the compatibility of codes. As WebAssembly and WebAudio are both young W3C standards with lots of experimental features, few implementations were made in open-source community are available to follow. We are facing multiples challenges in this project.
We hope that this documentation could push these new technologies further.
Faust is an audio processing programming language developing by Grame-CNCM and Stanford. Thanks to LLVM and Emscripten, the Faust compiler originally written in C++ could be ported to WebAssembly module easily. When the compiler was casted in a WebAssembly file libfaust-wasm.wasm, a corresponding JavaScript file is also been generated as JavaScript frontend libfaust-wasm.js, including an API which allows us to access the WebAssembly-side functions and variables with JavaScript. This frontend uses "Universal Module Definition" aka. UMD which friendly exports the whole wrapper as a compatible JavaScript module in widely-used JavaScript environment.
Files llvm-dsp.h and libfaust.h define all the functions available in libfaust. In addition, The Emscripten module serves C-like methods like _malloc or _free to interact with variable pointers. These are useful to set parameters or get results of libfaust functions.
Generally, the resulting code should be able to take Faust codes and some compilation parameters as inputs, then give an WebAudio Node as output. In other words, it is a WebAudio Node generator for other web applications that uses these nodes. Then the best is to consider and compress everything in the project into one single JavaScript file as an UMD module for anyone needs to build a web app to load.
In the past versions of this project (webaudio-wasm-wrapper.js written by Stéphane Letz), we already have the most part of functions that worked. But we need to clean them up then transform them into one JavaScript module. Here is a list of existing functions:
- Load libfaust and import its C functions into JavaScript
- Compile code: input Faust codes as a string, output its WebAssembly binary version, and some related data.
- load the binary processing module inside an AudioWorkletProcessor or a ScriptProcessor. (mono and poly version)
The project uses modern JavaScript tool-chain for a better maintainability and compatibility, even some of features in tool-chain are experimental, we don't want to sacrifice the simpleness of deployment of JavaScript module system. Here We list the technologies we used and the reason that we chose them.
As we are developing a web application using WebAudio API, obsolete web browsers will clearly not be able to run it. Even though, we cannot predict the browsers and versions users are having, and modern browsers implement Web APIs differently, cross-browser compatibility issues permanently annoyed developers for last decades. Fortunately, some solutions are more and more reliable recent years, then became a widely used standard.
Babel is currently one of the best solution to convert modern JavaScript languages into a compatible version of JavaScript. Using Babel tool-chain, developers can be free to write code in any version of supported language; by defining a list of browser versions, Babel will automatically transform the syntax or use Polyfills (equivalent functions) to generate corresponding JavaScript codes for target browsers.
We choose TypeScript as our main language especially for maintainability reasons. Traditional ECMAScript without typings makes developing IDEs difficult to hint codes or to predict issues. We will need to reread existing code to remember the types of variables and parameters which is a waste of time. Solutions like JSDoc is neither a perfect one since IDEs are not able to correctly parse all of them, and it doesn't participate the final compiling phase. TypeScript, maintained by Microsoft is actually a solid and common solution to deal with these problems. In addition, Babel now fully support TypeScript, which makes this tool-chain a nearly perfect combination of code-writing and pre-compiling.
.babelrc
In this file we define configs for Babel. firstly a list of Browsers ued by
babel-preset-envwhich will decide the syntax of the compiling result. Here we decided to support main stream browsers from version that support ECMAScriptclasskeyword, as WebAudio WorkletProcessor needs it. Some Polyfills are used to transform async functions, class properties, object rest spreads.
tsconfig.json
In this file we defines configs for TypeScript compiler. Some of options are not important as Babel ignores them, but option like
libneed to be defined carefully as it will affect code hinting and pre-compilation.
Linter is a tool that analyse potential programming errors or typos, we are using linter to formalise coding rules to keep the project clean and uniformed.
Since version 0.4.9, we no longer uses both TSLint and ESLint, as ESLint includes recently supports for TypeScipt through @typescript-eslint plugin. Configs are listed in .eslintrc.json file. We are actually using a coding rule based on airbnb-base.
VSCode integrate ESLint for analyse codes in real-time which shows all problems in real-time when coding.
As we need one single JavaScript module file as result, Webpack can help us to manage all the dependencies in the project and combine them together. Webpack now works perfectly with TypeScript and Babel by adding the babel-loader.
Webpack reads webpack.config.js as its configuration. Two modes of packing development and production are available for including the source-map which facilitate the debugging in browsers, or generating a minified version of codes which will improve the performance.
A main challenge we are facing here is to include the *.wasm WebAssembly module also into the bundle. By using url-loader of Webpack, it encodes the *.wasm file as DataURI strings which could be imported with JavaScript.
However it is tricky to import them with TypeScript. We added a TypeScript Definition File wasm.d.ts where we declare a module named *.wasm, then TypeScript allows us to write imports below:
import libFaustDataURI from "./wasm/libfaust-wasm.wasm";
import mixer32DataURI from "./wasm/mixer32.wasm";Here, libFaustDataURI and mixer32DataURI are DataURI strings.
After version 0.4.9, libfaut-wasm.data file comes along with libfaut-wasm.wasm. This file includes Faust standard library packaged by Emscripten. Libfaust will load it on startup by fetching this file. As we use url-loader instead to bundle the *.data into one single file, we are overriding the locateFile function in libfaust JavaScript wrapper to fetch the file by its DataURI, same for the *wasm file.
JavaScript wrapper of Emscripten module needs to fetch the *.wasm file while initiating itself, so the process is not a synchronised function. We need to know when it is ready before we use it. Actually the module has a then() function that we can use to set a callback while the module is ready. Unfortunately, it returns the module itself which is a "thenable" object that JavaScript will call its then() function. It causes an infinite loop. (See issue #5820)
Some workarounds can be found below the issue. We are using the same idea that to delete the then() function after it has been called. In the main Faust class, user can use ready getter to access the fixed Promise.
A brand new feature of WebAudio API, AudioWorklet is currently implemented no where but Chromium. It provides possibility to run an audio processor in a dedicated thread that not affected by other JavaScript code execution. Thus it is a unavoidable substitution of the old ScriptProcessor which is fundamentally flawed as it runs the audio process in the main thread.
Using Audio Worklet consists of two parts: AudioWorkletProcessor and AudioWorkletNode which represent handlers in main thread and the audio processing thread.
TL;DR, we will need to add the module by calling
((audioCtx as AudioContext).audioWorklet as AudioWorklet).addModule(("processors.js" as string));where we need a string that represent the name of the file of AudioWorkletProcessor. However, as Faust generates the processor dynamically, the processor file is different each time.
In the FaustAudioWorkletProcessor.ts, we have a static structure of the processor, the dynamic part will be injected into the scope which is some WebAssembly modules as base64 strings. We need to ensure that the structure is valid standalone by configuring Babel browser list, and that it can be executed while AudioWorklet get the file. The way we did is to cast the static structure in a function, then use Function.toString() method in order to inject the dynamic part. Finally the whole string will be transformed to a Blob URL for the parameter of AudioWorklet.addModule.
Update
Since version 0.4.9, we realised that WebAssembly.Module is a serialisable object that can be passing into AudioWorkletProcessor Scope by adding a parameter in AudioWorkletNode constructor.
So we no longer inject base64-encoded WebAssembly Module into AudioWorkletProcessor. However, static data like processor id and DSP metadata cannot be passed in this way as they should be known before the construction of AudioWorkletProcessor, especially for getting AudioParams.
Emscripten uses by default a virtual internal file system in memory which facilitates us to get the files generated by libfaust. We are using this feature to access the *.svg diagrams generated.
The system becomes useful as we includes full Faust standard library in the module. So we are exposing it by a getter in Faust main class.
We added recently a plotHandler callback into the DSP node. This allows additional processing or analysis after the buffer was fully calculated. The callback returns current output buffer, current buffer index and parameters change events.
In addition, to be able to calculate audio separately with a Faust DSP independent from the browser audio context, we created a FaustOfflineProcessor which will be used exclusively for getting the very first samples calculated by a DSP, aka faust2plot. This allows us to debug a DSP with a different sample rate.