Skip to content

Latest commit

 

History

History
64 lines (38 loc) · 5.84 KB

why-focus-on-ggml-and-ncnn-on-android.md

File metadata and controls

64 lines (38 loc) · 5.84 KB
Raw

1.GGML is a very compact/highly optimization pure C/C++ machine learning library. GGML is also the solid cornerstone of the amazing whisper.cpp and the magic llama.cpp. Compared to some well-known machine learning frameworks/libraries (e.g. Google TensorFlow, Microsoft ONNX, Meta PyTorch, Baidu PaddlePaddle......), GGML does not have much/complex/complicated/redundant/… encapsulation, so it's very very very useful/helpful/educational for AI beginners. In general, GGML has following features:

  • Written in C
  • 16-bit float support
  • Integer quantization support (4-bit, 5-bit, 8-bit, etc.)
  • Automatic differentiation
  • ADAM and L-BFGS optimizers
  • Optimized for Apple Silicon
  • On x86 architectures utilizes AVX / AVX2 intrinsics
  • On ppc64 architectures utilizes VSX intrinsics
  • No third-party dependencies
  • Zero memory allocations during runtime
  • all in one source file and similar to imgui(this is just personal opinion and I really like it, this is a NEW coding style and the requirements for programmer is relatively high and very helpful for experienced programmer, this coding style may not be acceptable in large commercial IT companies because it violates some principles of modern software engineering)

There are four "killer/heavyweight" open-source AI application engines based on GGML:

  1. A successful PoC illustrates/proves that ggml can be used in real(or commerical) AI application on Android device or other edge device.

  2. NCNN is a high-performance neural network inference computing framework optimized for mobile platforms. NCNN is deeply considerate about deployment and uses on mobile phones from the beginning of design.NCNN does not have third-party dependencies. It is cross-platform and runs faster than all known open-source frameworks on mobile phone cpu. Developers can easily deploy deep learning algorithm models to the mobile platform by using efficient ncnn implementation, creating intelligent APPs, and bringing artificial intelligence to your fingertips. NCNN is currently being used in many Tencent applications with 1.3b+ users, such as QQ, Qzone, WeChat, Pitu, ... and as well known Tencent is one of the most advanced/powerful IT gaints in China/our planet. so it's very very very useful/helpful/practical for commerical programmers/developers/applications. In general, NCNN has following feaures:

  • Supports convolutional neural networks, supports multiple input and multi-branch structure, can calculate part of the branch
  • No third-party library dependencies
  • Pure C++ implementation, cross-platform, supports Android, iOS and so on
  • ARM NEON assembly level of careful optimization, calculation speed is extremely high
  • Sophisticated memory management and data structure design, very low memory footprint
  • Supports multi-core parallel computing acceleration, ARM big.LITTLE CPU scheduling optimization
  • Supports GPU acceleration via the next-generation low-overhead Vulkan API
  • Extensible model design, supports 8bit quantization and half-precision floating point storage, can import caffe/pytorch/mxnet/onnx/darknet/keras/tensorflow(mlir) models
  • Support direct memory zero copy reference load network model
  • Can be registered with custom layer implementation and extended

There are some "killer/heavyweight" open-source AI application engines based on NCNN:

  • ASR(Automatic Speech Recognition, or Audio2Text)sherpa-ncnn
  • ......

  1. GGML and NCNN both support Android & iOS which maintained their position as the leading mobile operating systems worldwide with a market share of 90+ percent. Android phone could be seen as a standard embedding development board/device and it's available everywhere.

  2. Comparison between the two excellent AI edge inference framework.

  3. AI model trainning and finetuning will be done by AI experts/algorithm engineers. this project is a workbench for learing&practising AI tech in real scenario on Android device, focus on AI inference and AI application on Android.

  4. GGML based command line application(which provided in upstream ggml/whispercpp/llamacpp) works fine/well as expected on Linux/Windows/Mac, but it might be not work properly as expected on Android because Android phone could be seen as a special/standard embedding development board. A dedicated ggml/whispercpp/llamacpp derived project focus on Android might be useful for community.

  5. Android maintained its position as the leading mobile operating system worldwide in the fourth quarter of 2023 with a market share of 70.1 percent.

  6. I'm good at Android software developent(from UI to framework to kernel) and know a few things about iOS application software development although I know nothing(or a little) about real/hardcore AI tech. I heard whisper.cpp(used for generate subtitle with some English short videos in my Wechat public account) since mid-January 2024 and started learning ggml since 03-05(March 5) 2024 and ggml was born/open-sourced on 2022, I heard ncnn since end of April 2024 and started learning ncnn since mid-May 2024 and ncnn was born/open-sourced on 2017. my resource and capability is limited.