-
-
Notifications
You must be signed in to change notification settings - Fork 3
FAQ
This is an AAP documentation, explained in FAQ style.
AAP is to realize an audio plugin format as well as its audio plugins ecosystem on Android platform.
People use DAWs (Digital Audio Workstation) to compose music, or record live audio performances, and so on. Audio plugins are plugins for DAWs to work as either instruments or effects (or sometimes otherwise). As long as an audio plugin software conforms to an audio plugin format, which is usually some sort of API, it can be used by any DAW that supports the format.
There are private audio plugins that are only for specific DAW(s), and publicly shared audio plugins that can be used by any DAWs that support it. AAP is latter.
On desktop PC world, there are some popular audio plugin formats:
- VST(2 or 3) by Steinberg
- AudioUnit (AU, v2 or v3) by Apple
- CLAP by Bitwig
- Some community driven plugin formats like LV2 (popular on Linux) and WebAudioModules (Web)
AAP is the only audio plugin format on Android (I'd say). The reason why Android does not have any audio plugin format is that those traditional ones (VST2/3, AUv2, LV2, CLAP, ...) are designed to work only on desktop OSes (we mention more about it at the "Rationale" section). Apple's AUv3 is the only exception; it is designed to work on iOS. AAP stands at similar position to AUv3.
(There used to be a similar project to AAP called AudioRoute, and it has a supported DAW called n-Track Studio. But the development activity on GitHub has been stopped since 2021.)
There is no audio plugin format that can be used on Android. That means, all instrument plugins and effect plugins on Android are specific to certain DAWs. We want to change that, and have Android as a viable platform for audio and music creatives.
AAP does not aim to just be a plugin format in-theory, but with the actual plugins ecosystem to use, to avoid typical "chicken and egg" problem on a plugin format. If your DAW or other audio app supports AAP, there will be already 20+ plugin apps with >100 plugins that are ported from cross-platform audio plugins written in JUCE or LV2.
AAP is still under heavy development, and also lacks viable products as an audio plugin format. However, things are partially working.
Most notable one would be AudioPluginMidiDeviceService. You can turn any AAP instrument plugin into Android MidiDeviceService using AAP. It would be impressive as you can use instruments like Sfizz, Fluidsynth, OB-Xd, Dexed, ADLplug, Odin2, Vitaloid (my Android port of Vital OSS) etc. as MIDI instruments. Install those AAP ports, and launch your favorite MIDI keyboard app or DAW that supports MIDI output. You can find new MIDI output devices for those instrument plugins. (Do not try JUCE based Android apps, like ZenBeats. JUCE Android MIDI support is completely broken.)
There is no one I believe. Note that AAP is not practically usable yet.
Expecting community while there is no user...? Well, there could be developer community. Though again, no there isn't yet. @atsushieno is the only developer so far. The GitHub repositories are the official places that handles bug reports (issues), development (pull requests), and discussions. We always welcome contribution, helps and joining development.
However there are handful of people who told @atsushieno that they are interested in AAP. @atsushieno is around some audio dev. community and would answer to questions there too. See the user profile to find those.
One of my dream is to have generative music platform, where audio plugins work on any environment and platform. Audio plugins are essential to music authoring. Missing audio plugin format on Android is definitely no-go.
I should be making money and make AAP ecosystem sustainable, but so far everything is under development and the ecosystem is not at very usable state.
Note that ports do not mean they are usable on all features the original projects (LV2 or JUCE) offer (most notably, there is no GUI ports in AAP context, while those JUCE standalone apps run their UI: juce_gui_basics does not implement Android UI embedding).
It is not practically usable, but you can experience my AAP integration to Helio Workstation, a JUCE-based DAW: aap-juce-helio
Tracktion Engine is also known to build on Android, so there are more potential.
Not yet. I don't plan to publish useless apps in general and collect a lot of 1-star reviews.
We have a dedicated installer application for AAP: AAP APK Installer
It downloads those plugin APKs from either a GitHub release or a build artifact. It is developed as an OSS, and the build artifacts are available on GitHub Actions. Everything is automated and safe as much as you can read the source code. See the GitHub repo for details.
Technically we can, but that would not meet the ideal state of the plugin ecosystem: everyone can publish their plugin or DAW by themselves, not dominated by us. It is just like AUv3 plugins on iOS App Store (or Mac App Store).
The entire AAP ecosystem is not a single application, but a bunch of applications.
One or more AAP plugins can be packaged and distributed as an application (apk/aab). For example, AAP MDA-LV2 is distributed as an apk and contains 35-ish plugins.
Isn't AAP just a specification i.e. an API definition? Why does AAP have to be offered as the implementation?
Before giving the actual answer, we would like to warn you - you should probably skip learning about the API in AAP, if you are just a plugin developer, or a DAW developer even. We don't intend to let you learn about it yet.
AAP is certainly a specification on one hand, as it defines a set of API in C header files. At the same time, the plugins has to exist as an AudioPluginService which is an Android Service.
We offer androidaudioplugin.aar that contains the AudioPluginService class as well as the hosting implementation, along with C/C++ headers and implementation.
The plugin hosts (DAWs) have to function as Service clients. In theory they could be implemented totally independent of androidaudioplugin.aar, but both are bundled in this single aar so far.
There could be better isolation and we continuously explore that.
Sometimes people guess it is (as we mention JUCE a lot). No. we support JUCE apps (by providing custom audio plugin format implementation for juce_audio_plugin_client as well as plugin wrapper implementation for juce_audio_processors), but AAP itself is not a JUCE application. AAP must be available under the MIT license so it will not be a JUCE application (unless their licensing model changes).
Having said that, AAP had started as an experimental JUCE project and it owes a lot of API design from it.
Our JUCE support is split into separate GitHub repository called aap-juce.
JUCE supports Android in general. And it would support any audio plugin format that exists. There is no such a format on Android yet, thus there is no plugin support. Note that JUCE itself is not an audio plugin format.
A JUCE audio plugin application could be built as a Standalone module, and we make use of it on Android.
AAP supports LV2 in that LV2 plugins could be imported to AAP ecosystem. AAP is NOT an LV2 application in that we do not use LV2 as the underlying plugin format. aap-lv2 is certainly an LV2 application in that we use lilv (a modified version by us) for wrapping LV2 plugins built for Android.
When we want to "port" some existing audio plugin to another plugin format, we use "wrapper" technique (lv2vst, yabridge, Ildaeil, clap-wrapper); lots of the plugin APIs overlap their features, so they could be translated to work as if the wrapped plugin were implemented in the wrapping plugin API. aap-lv2 offers such a wrapper foundation. It is not a perfect solution, but it covers basic features.
We just haven't investigated it. That means, we are not sure how much we could offer for LV2 hosting that wraps AAP, and gain any benefit. We are not aware of any DAW that supports LV2 and runs on Android (unlike JUCE apps like Helio Workstation). It does not seem easy to port plugin hosts like Carla to Android; @atsushieno had a quick look and quickly gave up.
aap-juce brings in more problems than aap-lv2. But starting JUCE 7.0, LV2 is supported. Can't we just unify everything into aap-lv2?
The answer is NO - aap-lv2 is more or less a wrapper solution that makes use of LV2 SDK (serd/sord/lilv) and the actual plugin implementation may bring in their own problems and limitations. JUCE brings in the same problems and limitations that aap-juce addresses (or tries to address). In the end, "aap-lv2 for JUCE" will result in what current aap-juce looks like.
AAP does support extensions, and it should offer extension features to preserve API compatibility. Though we keep breaking the core API and it would take a while until we finalize the API.
Also, due to the nature of host-plugin relationship in isolated processes, our extension API actually involves some implementation code. See Technical Details section to learn more about "AAPXS".
AAP as an audio plugin format does not depend on Oboe. It is quite common to any audio plugin format, even including their implementation SDKs. It is because audio I/O and MIDI I/O are what DAWs deal with. Audio plugin SDKs usually deal only with binary memory stream for both audio and MIDI data. That's why typical audio plugin format API is cross platform (until they deal with GUI).
AAP default plugin manager (androidaudioplugin-manager.aar) and MidiDeviceService (androidaudioplugin-midi-device-service.aar) are different. For instance, the latter works as a MIDI output device, which generates audio outputs. Thus we make full use of Oboe there.
AAP intends to offer audio bus configuration extension, but so far it is not implemented yet.
Yes, probably more than what you expect - AAP deals with MIDI events in MIDI 2.0 UMPs.
MIDI 1.0 stream could be translated to UMPs according to MIDI 2.0 UMP specification. You can use atsushieno/cmidi2 library that supports UMP-MIDI1 bytestream translation. JUCE also provides UMP translators.
Do you mean, AAP "maps" and consumes MIDI events for its own? Should we worry about some messages swallowed by AAP?
UMP is the event message structure. Parameter changes are sent as Universal SysEx8 messages, or optionally Control Changes, or Assignable Controllers (NRPNs). Presets can be selected by Program Changes too.
Unlike VST3 MIDI CCs and program changes, those mappings can be disabled by each plugin so that it can receive raw UMP message as is.
The answer is going to be complicated but in short: not now. Because it is impossible. See Technical Details section on why.
Having said that, AAP is designed to be really close to realtime safe.
Since early 2023 AAP has preliminary GUI support in two ways so far:
- Web UI through
WebViewandJavaScriptInterface: host instantiates plugin UI, the UI controls the plugin remotely. - Native UI through
SurfaceControlViewHost: plugin instantiates the UI through host's requests (via GUI extension). It makes use of Android 11 API. We offer our default native UI implementation based on Jetpack Compose.
See docs/design/GUI.md for details.
Note that usually audio plugin GUIs that were developed for desktop are awful on mobile platforms. You cannot really control those small knobs using fingers (your finger will hide them from your sight) and every control will be coarse. Their UI should be optimal for different form factors.
There should be many things we could learn from AudioUnit V3 on iOS, not from those desktop plugin formats.
Looks like there are many JUCE plugin ports to AAP. Can we use their GUI on AAP as the "native UI" ?
Sadly the answer is NO. JUCE GUI is usually implemented using juce_gui_basics module, and its Android support works like that (1) it always works as JuceActivity, and (2) the only Activity that JUCE application runs is JuceActivity even if you specify otherwise on AndroidManifest.xml (JUCE initializer invalidates the main launcher activity and switch the current activity to its own).
To make JUCE GUI working, we will have to make a lot of changes to JUCE, which will result in some fork that is hard to maintain. It is still an option, considering that we were planning to use juce_emscripten for Web UI integration, but @atsushieno will not have engineering resource enough to work on it.
People are tempted to discuss like that. But none of those who state like that actually provides solution for mobile and web platforms. How can VST3, LV2, CLAP work on Android as is? You cannot load plugin's dynamic library in the host process as it is prohibited. Then your only solution with those existing formats is to publish both plugins and DAWs within one single application. It is as if only Steinberg shipped ALL audio plugins under the name of Cubasis. No one would be able to ship their plugin without approval from Steinberg. That's not a diverse ecosystem.
Apple came up with their complete answer: AUv3. It is a different plugin format than AUv2. They are incompatible. AUv3 is not a privileged solution on iOS - except for GUI overlay etc. It has to deal with isolated processes from the ground, at the cost from isolated processes. People often find it awkward, but that had to be done. It's like we cannot go back to HTTP without S. AAP is at a similar position.
IPCs bring in different design considerations compared to mere function calls/callbacks. For example, CLAP "threadpool" extension assumes that calling back and forth between host and client within realtime processing costs almost nothing. But think about the cost of those roundtrip calls on mobile platforms - they cost a lot. Basically AAP considers there should be only one IPC call for each audio processing call (details mentioned later).
Likewise, WebAudioModules would never happened if any of the existing plugin formats worked.
In practice, when people say "we should not invent another audio plugin format", there would be three kinds of people. Here are the answers to each:
- "We don't want to build for multiple plugin formats unnecessarily" => it does not apply if there is no alternative format. AAP is so far the only one solution for Android. You build AAP, or you don't support Android.
- "We don't want to write code for another plugin format" => people already don't. They use multi-plugin-format exporters like JUCE, DPF, FAUST, MATLAB etc. See how CLAP plugins have been built. We offer entry slot for JUCE and LV2 (thus indirectly DPF too). You can use them to program your plugin, then build the AAP version using aap-juce or aap-lv2.
- "We don't want to depend on a plugin format that may vanish and the plugin binaries became useless" => true, though plugins with sources lived longer, and binaries would not die that soon as long as the DAWs and the plugins remain on the platform. On mobile OSes you should probably rather worry about apps vanishing from the platform due to the platform updates. arm32 is already dead, and Android 14 will be arm64-only on Arm architectures.
Considering that AAP is at similar position to AUv3, maybe? But note that AU is the only plugin format that the platformer designed the API. (Well, Microsoft also had DirectX plugins in the past, in case you care.) Essentially, anyone could work on such a project.
Because they are "the top two". As far as I have searched around GitHub repositories, JUCE is the most used audio plugin SDK by the audio plugin projects out there. LV2 is next. Of course they live in different tiers (JUCE is not a plugin format and LV2 specification is not an SDK), but from our (plugin porters') point of view, these two are the most beneficial.
DPF probably comes next. But since we already support LV2, DPF is already covered via LV2. We do have aap-lv2-dragonfly-reverb, aap-lv2-string-machine, and aap-lv2-aida-x.
There are much less audio plugin projects that are implemented in direct VST3 API. So as CLAP. While there are many people who find that CLAP is a future, they do not really implement CLAP plugins in CLAP API. (Their community has been actively trying to make it realize with clap-wrappers.)
Having said that, we are still seeking for CLAP integration on occasion, for host developers' sake. CLAP is one of the best suited plugin solutions for embedding to host apps.
I figured that everything is so hacky. I don't think it is a good approach. Why don't you build a solid foundation first?
@atsushieno did not start the project as a perfect C/C++ audio developer and did not know much about modern C++. Probably still not. It should also be noted that it was not started with any solid plan. It was a hobby bonsai project which grew randomly.
But that doesn't matter. What is the most important in our perspective is to show a blueprint of the entire plugin ecosystem that looks feasible, which is a grand ecosystem with a lot of plugins. Those plugins were ported to explore many different porting setup (JUCE plugins are quite diverse; they use either CMake or Projucer. Plugin can be an effect or an instrument. JUCE version difference matters) which helped designing the entire ecosystem.
Crappy implementation internals can be changed at any time while the iron is hot. Designing portable plugin ecosystem takes another significant steps. The AAP fundamentals have been reflecting the porting experiences.
and lastly, @atsushieno has some updates on this: at v0.7.6 it had become way more stable than before.
One of the latest trends on certain operating systems (like Android, iOS, or macOS), often along with the application markets, is that each application is "contained" and has explicitly limited access to the computer resources. They cannot infect on the platform e.g. they have no access to system file resources or files on other apps' space, or they cannot dynamically load (dlopen(), LoadLibrary()) other app's native executable code.
On desktop platforms, a plugin host (DAW) "instantiates" an audio plugin by loading its executable code (dynaic library). It is prohibited on modern OSes as explained above. Therefore, every interactive operation needs to be performed through some kind of IPC (inter-process communication).
AAP depends on Binder IPC and shared memory. The audio processing IPC performs without copying audio and MIDI buffers (because they are on the shared memory) so if the IPC stack works in realtime manner it could work in realtime.
Binder is a low level IPC mechanism that resides partly in Linux Kernel internals. It is special in that the messaging involves shared memory space to send and receive arbitrary data structure (to some extent) within a machine, and involves native threads, often realtime ready, that can switch tasks seamlessly.
AIDL is the interface definition language. We have our AIDL for AudioPluginService. A host and a plugin interact each other based on this protocol.
Whenever our AIDL changes, the entire ecosystem gets broken and the world gets diverged between old and new. We should avoid that at all cost (but things happen until we finalize the API). We do not intend to let DAW developers and plugin developers to deal with the binder messaging directly. They are capsuled in the API in androidaudioplugin module (both C/C++ and Kotlin).
If you are an Android app developer, you would have heard like "Use WorkManager for most cases". It is probably true for Java/Kotlin app developers. We are different. We are native C/C++ developers who worry about realtime safety and performance.
Audio processing has to be done in very short cycle like <10 milliseconds for professionals. Our requirement would not be that tight, but it should not involve extraneous latency caused by the Dalvik bytecode VM (ART, for Java/Kotlin code).
As explained in the next answer to the Q, binder is capable of realtime processing at its core. Binder support is still actively developed in Android platform, and it has public native API since API Level 29, which is still somewhat new.
In short: Binder itself is, but we cannot benefit from it, at least yet.
More sophisticated answer:
Binder after Android 8.0 is implemented to work in realtime manner. However, Binder does not support realtime processing for general application, while it is technically possible(!). There is a bit more detailed description on AOSP website. Their audio related services could benefit from it, but our apps that depend on framework /dev/binder cannot.
When a Binder IPC is not realtime safe, it involves a lot of mutexes, which means no-go for realtime audio.
We could discuss with Google to open some opportunity to it. But we would need more PoC product first.
One of the implicit principles in AAP is "whenever a standard exists and is useful, prefer using it". MIDI 1.0 was too rough for modern audio plugin processing, but MIDI 2.0 UMP (Universal MIDI Packet) is fairly good. Many UMP message comes with 32-bit data, namely in Control Changes, Registered and Assignable Controllers (RPNs and NRPNs in MIDI 1.0), key pressures (both channel and poly), and Pitch Bend. Ultimately we use System Exclusive messages for controlling parameters, but being capable of processing UMPs directly, we would not even need MIDI mappings that each DAW somehow manages (sometimes even plugin format involves; CLAP).
AAPXS stands for "AAP eXtension Service". Unlike typical desktop audio plugin formats, we have to deal with some sort of mediator between host and plugin. AudioPluginService is one part of it.
Audio plugin extensibility is usually achieved in the form of dynamically loadable extension functions. But as we have discussed earlier, Android host cannot load plugin code dynamically, as well as plugins are unable to invoke host functions by pointer. Therefore, as explained earlier, we use IPC instead.
Extensions have to be implemented like binder messages. It is based on extension() service method in our AudioPluginService AIDL, and it only passes shared memory for request body and response body, indicated by the "opcode". We serialize and deserialize the request and the response on both sides.
AAPXS involves some implementation code, unlike extension API in other plugin formats, to implement these serialization parts that every plugin developer and host developer should not deal with by themselves.
It feels like we are inventing another wheel for serialization format in AAPXS. It is true in a sense, but since it is not that complicated, it should not be of a lot of problem. It is rather problematic to incorporate protocol buffer library into libandroidaudioplugin.so.
AIDL is good for protocols where all plugins and hosts implement everything, or it is okay if we can implement different Services for each AIDL. Apparently the latter is not true for AAP - there should not be too many IPC connections. If we define all extensibility in AIDL, that would mean every DAW and plugin needs to update the implementation whenever a new extension API appears (and not all extensions are "official"). They would work without missing service methods support, but the AIDL will be kept updated frequently and it will mess implementing service or your binder proxy.
Why do we have to export plugin metadata as in XML? Couldn't you choose better data format like JSON?
The answer is twofolds. We provide plugin metadata as Android Service metadata. It can be queried by PackageManager.queryIntentServices(). It returns the XML metadata content that is referenced from AndroidManifest.xml without instantiating the plugin.
If we decided to let each AAP plugin to return metadata by code, it would have been required to instantiate each plugin (or plugin factory, like what CLAP does) and that prevents fast plugin scanning. VST3 had similar slow scanning problem until 3.7.6, and then it started to support metadata in JSON (opt-in feature).
And interestingly XML is the only format Android system allows to return the metadata content. We cannot choose JSON etc. for that reason. The acceptable content size seems to keep shrinking, so at some stage we might have to remove some parts from aap_metadata.xml (e.g. <parameters> are already optional and can be dynamically populated in code), but the basic policy to provide metadata with XML won't change.
A followup: Android 31 had introduced <resource> alongside <meta-data> in <service> declaration. Our minSdk is 29 (and 30 if you expect in-plugin-process GUI), so we keep using XML. Adopting JSON is not good enough to drop API Level < 31.
SurfaceControlViewHost is a (somewhat) new API in Android 11 that lets an application to serve its own UI hosted by another application. It makes use of SurfaceView at "client" side whose content can be transmitted via Binder connection. For more details, this blog post would help you understand it well.
Before we adopt this SurfaceControlViewHost, we implemented GUI based on SystemAlertWindow. But it is not a recommended practice and involves users' special permissions (like IME applications or Facebook Messenger chat head).