Kivy
Replies: 5 comments 3 replies
-
|
So then Build for Android and iOS will be very fast. Because packages are already complied. Great Idea. 😀 |
Beta Was this translation helpful? Give feedback.
-
|
I am so excited . To use that . 😀😀😀😀 |
Beta Was this translation helpful? Give feedback.
-
|
Nothing real to add, great change, seconded! |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
Yes
Thanks for suggesting, this should be automatically done while packages are being downloaded. |
Beta Was this translation helpful? Give feedback.
-
|
We need all possible way . To build fast build which is already compiled. And build by compiling source. Which gives full flexibility in build process |
Beta Was this translation helpful? Give feedback.
-
|
Did you study the wheel file format? Could it not possible to use it directly instead of making a new intermediate format? |
Beta Was this translation helpful? Give feedback.
-
|
We are using wheel files for python libraries only. But can we use wheel format for custom non-python libs also? |
Beta Was this translation helpful? Give feedback.
-
During the last few days, I've worked alongside @T-Dynamos (who is doing a great job in maintaining
python-for-androidand migrating recipes to isolated builds) to write this proposal.Feel free to share thoughts or send suggestions.
Preface
During the past few years, users were forced to build their dependencies from scratch, and that made quite un-easy for newcomers to hike the Mount Everest of “cross compiling”.
This issue does not only applies to non-plain-python python packages, but also (and not limited to) libraries such as
sdl2,libffiorsqlite3and our belovedpython3interpreter.What was basically forcing us (or making it harder) to avoid having pre-build artifacts is the inability to build python packages in an isolated mode.
However, thanks to a contributor (@Anshdadwal) awesome work, we’re now able (at least on python-for-android) to build python packages in an isolated environment.
Additionally, PEP 730 and PEP 738 will make things easier quite soon, so now seems the best time to start working on building and hosting pre-build artifacts.
The current status
Both
python-for-androidandkivy-iosare specifically made to being run on the developer machine, and even if testing is made on CI/CD side, we do not have a proper workflow to get the build and store artifacts, so mostly everything needs to be done from scratch.The proposal
Phase 1: Separating build from installation, and creating proper install archives
ATM all the produced artifacts are automatically installed. We do not have a clean separation between the build and the installation phase, even if we tried to do so in code (sometimes).
The build and the install part should be separated clearly, so:
archive format: A
.zipor.whl(for wheels, no need to re-invent the wheel 😅) file.The
.zipartifact should have the following folder structure:metadata/properties.jsonis a json file which stores all the package properties, such as:-
stage: aka thepython-for-androidbranch (master,develop, RFU (such as backports support for old python versions)-
name: extracted from recipe name - str-
arch: Options may differ based on the target platform (iOS or Android). In case ofiOS, it may even be a multi-arch label.-
native_api_level: minimum supported api version (ndk version for Android, min supported iOS version for iOS)-
kind: defined by recipe kindlib,python,files-
version: extracted from pinned package version in recipe - str-
release_version: RFU - str-
lib_dependencies: list, should be a validlib-
files_dependencies: list, should a validfilesartifact.metadata/install.jsonis a a json file which stores a list of install instructions, which are executed from top to bottom.Install instructions are composed of install primitives (ATM only
copyis available) and additional options:-
copy: Used to copy files from the artifact to the install folder.-
options:-
src: Mandatory, relative to artifactdatafolder-
dst: Mandatory, may contain placeholders such as$PROJECT_FOLDER,$ANDROID_LIB_FOLDER, etc. (todo: define placeholders)data: You can store basically anything into the data folder, and you can follow your own structure.archive name:
stage-(lib,python,files)-name-arch-native_api_level-version.zipDifferent artifact kinds are installed in the following order:
libspythonwheelsfilesOnce the CI/CD pipeline notices a change in the
recipes/namefolder the package is built and saved as an artifact result on Github PR, so we should define a common path to place built artifacts.Phase 2: Automated build. Manual upload. Manual test.
We now have proper storable and installable artifacts, but the build & install phase still happens on the same machine.
During this phase, we will allow users to download artifacts from a publicly available storage, however, the artifacts upload will be performed manually when a maintainer (or contributor with additional privileges) decides to perform a release. Only the
developstage will be available.Additionally, to avoid incurring in high-bandwidth requirements (and abuse), users should manually download and place files in a folder where
python-for-androidorkivy-ioscan later retrieve it. That means no automation is still included in both projects, but we will configure both to check for pre-built packages in a cache directory.This phase will allow us to perform tests (check that the builds effectively work as expected when used on a different machine, version check, dependency tree, etc …).
Phase 3: Automated build. Automated upload. Manual Test.
Time to build a “bot”, to handle releases. Artifacts are always created, but should be uploaded to the distribution storage only when a release is requested by someone with the proper permissions.
The current CI/CD pipeline rules are just not enough as we need to specify which artifact(s) should be uploaded.
To do so, inside every recipe folder, a
release.jsonfile will be added. This file will keep track of every recipe release,The
release.jsonfile will have the following content:Once the CI/CD pipeline notices a change in the
recipe/name/release.jsonafter a PR merge, the CI/CD should try to upload the artifact to the distribution server (the upload process will be similar to the one currently in use bykivy/kivy)A
stage.jsonfile will be added on the root folder of the project. That file will contain information regarding the current stage, which can be used during CI/CD operations or build.Phase 4: All automated.
Before this phase, we will work towards our current hosting sponsors or look out for new partnerships, to make sure the Kivy infrastructure can sustain continuous downloads from the pre-built package storage.
python-for-android, andkivy-ioswill be configured to look-up for pre-built packages from the selected stage.If no pre-built package is available, but a recipe is,
python-for-android(orkivy-ios) will try to self-build that package.About stages
Each stage will have its own endpoint, which is allocated manually upon request by a Kivy Core Developer team member (or by someone with the proper privileges to access the Kivy web infrastructure).
kivy-iosandpython-for-androidwill have two different environments, so stages and other details can be configured differently.The stage should reflect a valid branch of the
python-for-android(orkivy-ios) project, so can be maintained / freezed based on the development needs.Wheels and non-wheels will have two separate environments for the same stage.
Every stage will have a total of 3 public endpoints:
About version and build management
Users will have the opportunity to pin to a specific version, if available, of a specific dependency (lib, files or wheel).
As seen previously, version should reflect the version of the library / tool / wheel which is being built by the recipe.
However, mistakes can happen, and we may ask users to download and re-install a specific library (or wheel). To do that, the maintainer, should bump the
buildnumber of the relevantrelease.jsonfile, so a new release will be performed.On
python-for-android(orkivy-iosside), the look-up, download and install logic should check if the build version have changed, and act accordingly. The upgrade can be automated, or an interaction may be required from the user, based on the user provided options.About supported archs and api levels
Obviously, we can’t store and officially support all the combinations of archs and api levels.
Most of the users, even right now, are just using the default NDK target api level, or on iOS the default supported version.
So, even if the new structure and changes will allow users to self-build artifacts with a NDK api level different from the one which will be made available for direct download, we will not store or produce artifacts for these “officially unsupported” versions.
On Android, only artifacts for
armeabi-v7a,arm64-v8aandx86_64will be produced and stored, asx86artifacts are considered obsolete and a niche.Beta Was this translation helpful? Give feedback.
All reactions