The following page details the Sire build process. Sire is built using Miniconda, creating an isolated Conda environment into which Sire with be compiled and deployed. This avoids the need for external dependencies, since everything that is required is already available within the Conda ecosystem.
The main script used to compile Sire is
compile_sire.sh
on Linux and compile_sire.sh
on Windows. You shouldn't need to modify this file, other than updating
the MINICONDA_VERSION variable if needed. (Note that since we use
a specific Miniconda we only currently build Sire for Python version 3.7.)
The build_sire.py
Python script is called by the above shell scripts to perform the build.
It is within this file that external dependencies are installed, using Conda
where possible, and pip if a package is not yet available via the Anaconda
Cloud. Please use the conda-forge channel
as your first port of call, since this will help maximise compatibility between
packages. (We plan on eventually migrating to the conda-forge build platform
ourselves, so it is important that we can eventually depend on a single Conda
channel.) When choosing / updating package versions please choose the most
recent version that is supported on all operating systems that we build for,
i.e. linux-64, osx-64, and win-64. To check which versions are
compatible use the Anaconda Cloud, e.g. for the boost.
(Obviously you can use an older version of a package if the latest is broken,
or if dependencies can't be resolved.) Note that versions aren't just
important for Sire itself, but also any applications that are built on top
of Sire, such as BioSimSpace.
Please don't change version numbers without checking that applications that
depend on them still work. (For example, RDKit
is used by BioSimSpace, which depends on some of the same packages used
by Sire.)
We use Azure Pipelines for continuous integration and deployment. On Linux this is manage using Docker containers, with details for each stage of the build available in the docker directory. Note that the pipeline can be unreliable because of network timeouts, so if a build fails it is often worth trying to queue another build of the same commit from the online control panel. (Check any error messages first to see if it looks like a genuine issue.)
The sire-conda
directory contains all of the files that are used to build and deploy
the Sire Conda package during
the Azure Pipeline. The scripts will query the Sire Miniconda for the
versions of any external dependencies, then add them to the recipe
for our package. This means that any version changes that you make in the
build_sire.py
file will automatically be updated in the Conda package, i.e. you only need to
maintain dependency versions in a single location. The only thing that is
needed is to update the list of external Conda dependences in
update_recipe.sh
if a new dependency is added. (Note that you'll also need to manage the
versions of any dependencies of dependencies that are required at run time,
since Conda doesn't do a good job of managing this for you. For example, the
netcdf4 Conda package doesn't pin the version of libnetcdf that it
pulls in, we need to manually pin the version of this package in the run time
requirements of our Conda recipe. Currently we need to do this for libcblas,
libnetcdf and qt.) The Conda package version number is also
automatically inferred from the git commit (we use the tag as the version
and the number of commits since the tag as the build number). The Sire Conda
package is created by extracting all of the compiled Sire binary and library
files from the Miniconda installation, then uploading them to the Oracle Cloud
as an archive. These are then downloaded and unpacked into the correct location
during the Conda installation. See this
script to see how the archive of files is created and this
script to see how the same files are unpacked in place during the Conda install.
Currently we pin the version of conda-build to 3.17 due to regular
performance issues (hanging) with version 3.18, particularly on macOS. The
Linux version is pinned in the Dockerfile
whereas the macOS version is pinned in the Azure Pipeline
configuration.