darwin: change the SDK pattern, update the SDKs, and update source releases #346043

Packages propagated by the SDK need to use a stdenv that does not propagate anything. Otherwise, an infinite recursion will result when building those packages. For consistency, all source releases should use the bootstrapStdenv.

Adding the hook allows the deployment target to be changed without having to mess with the stdenv. The can also be propagated, which is useful for libraries that have a minimum deployment target higher than the default in nixpkgs. In that case, they can propagate the hook to ensure library users are not targeting an unsupported version.

The new Darwin SDK pattern relies on an effectively empty, stub libc implementation. The actual libSystem to be linked is located dynamically based on the active SDK for the target. Independent build, host, and target SDKs are all supported by Darwin. The stub libSystem contains empty `include` and `lib` folders to avoid warnings from wrappers that add those paths unconditionally, which can turn into errors when a package is building with warnings-as-errors. While it would be nice if a fallback libc could be provided, SDK headers are not compatible between framework versions. Providing a fallback risks mixing headers from different frameworks, which can result in hard to diagnose errors involving semicolons or other punctuation.

This hook is used by source releases that build with Meson to assert that the Xcode project has not changed since the previous release. This is meant to be a check to force those updating source release packages to make sure they have incorporated any changes that were made to the Xcode project into the Meson build.

This is a replacement for the family of `appleDerivation` functions currently used. It is patterned after the `mkDerivation` used in the BSD package sets. It also provides additional support for using Meson to build source releases.

Add a new update script that locks the source releases to the indicated macOS version per the version pins in following repo: https://github.com/apple-oss-distributions/distribution-macOS

libsbuf is required by some of the source release updates that will be done. Unfortunately, it is only available on macOS 14 and newer, and there is no source release available currently. This is a port of libsbuf from FreeBSD, which appears to be the origin of the header provided in the 14.x SDK. It provides the same ABI as the system dylib and same API as the the SDK header while being available on all supported deployment targets in nixpkgs. Note: This package is not based on libsbuf from the FreeBSD package set in nixpkgs because: it doesn’t build on Darwin, and using it would pull many FreeBSD packages into the Darwin bootstrap, which is undesirable.

This is a new packaging of the Darwin SDK. Instead of splitting libraries and frameworks into separate packages, it provides a single package for the whole SDK. # Features - Vendored files are removed from the SDK. There are 50+ different packages that are vendored by upstream (depending on the version); - Components that are built in nixpkgs (either from upstream or from the source releases) are also removed. If they need to be included by default, they are propagated; - A single SDK pattern is used to package all SDKs, and scripts are provided to aid updating the SDK version and its source release versions. This makes adding new SDKs much easier; - SDK overrides are handled by adding the SDK version you require. If multiple SDKs are present, only the newest is used. It is possible to have different SDKs for each of build, host, and target platforms; - Private headers are no longer provided by default unless you use the SDK’s `privateFrameworksHook` to add them. It does the right thing when multiple SDKs are in your inputs; - Source releases for the SDK version are available via a passthru `sourceRelease` function. This is mostly useful for getting private headers for building source releases in the darwin attrset; and - The same versions of propagated components are used on both platforms (e.g., the same libresult, libiconv, etc). See `pkgs/by-name/ap/apple-sdk/README.md` for details on how the SDK derivation is structured and how to update it.

Some compilers may know to check these paths when `SDKROOT` is set, but it’s not assumed they do. `SDKROOT` is instead derived from the `DEVELOPER_DIR`, and `NIX_CFLAGS_COMPILE` is set up with the sysroot and necessary framework and include search paths.

… set It is unusual to invoke a wrapped compiler without setting `DEVELOPER_DIR`, but it can happen when a user adds a compiler to their installed packages or when a package intentionally invokes the compiler without an environment (such as the GHC binary packages).

Without this change, all Darwin platforms mangle to the same suffix salt. That is normally not an issue because build = host should mean a non-cross build, but it causes issues on Darwin with static builds because `DEVELOPER_DIR_FOR_BUILD` and `DEVELOPER_DIR` will refer to different SDKs but mangle to the same `DEVELOPER_DIR` with suffix salt. The fix is to mangle static builds differently from non-static ones on Darwin, which allows building for a static Darwin target on a same-architecture Darwin host. This fix is applied only to Dariwn because the issue does not appear to affect other platforms.

With this workaround enabled, CMake looks for SDK headers and libraries in `${SDKROOT}/var/empty`, which ensures it will not find them. Disabling the workaround in just this case allows CMake’s SDK support to work with the SDKs in nixpkgs while still not including other, impure paths in its search paths.

gnutar’s `configure` detects libiconv but fails to link it on Darwin.

Using the bootstrap stdenv avoids an infinite recursion from xcbuild depending on libpng depending on xcrun from xcbuild, which is propagated by the non-bootstrap stdenv.

The Darwin SDK used to provide `libutil.tbd`, but it has been replaced by `libutil.dylib` in the darwin.libutil package.

Using the bootstrap stdenv avoids an infinite recursion from xcbuild depending on libxml2 depending on xcrun from xcbuild, which is propagated by the non-bootstrap stdenv.

Newer SDKs use this on Darwin. Without it, using the 14.4 and 15.0 SDKs won’t work with clang 16.

While the Darwin stdenv bootstrap sets up its own clang wrappers and doesn’t provide these wrappers in the final stdenv, it does use them indirectly via the LLVM bootstrap to build LLVM and its libraries. Note on using the system libunwind: It is possible to build and use the LLVM libunwind on Darwin, but using the system by default one ensures everything is using the same unwinder.

Instead of using overrides in the stdenv bootstrap, Darwin will be relying on the LLVM bootstrap to build compiler-rt. The only special handling it needs is to use a stdenv with a bootstrap SDK instead of the default one (to avoid infinite recursions).

Also enable TSAN since the SDK has xpc. There’s no reason to disable it.

compiler-rt supports specifying the SDK path and version, so do that to avoid needing to include `xcrun` as a native build input, which simplifies the bootstrap.

The first build of compiler-rt in the LLVM bootstrap is build without libc++ being available, which causes support for the `-g` flag to be detected incorrectly on Darwin. Overriding the check by specifying that it’s usable allows the first build of compiler-rt to succeed.

When compiler-rt targets Darwin, it is built with `-target`, which causes clang to try to invoke `ld` without a target prefix (e.g., it will try to exec `ld` instead of `x86_64-apple-darwin-ld` on a cross-build to x86_64-darwin). Specifying `--ld-path` overrides that behavior, allowing it to find the appropriate cross-linker.

Using a bootstrap stdenv is needed to avoid infinite recursions that occur due to inputs propagated by SDK that depend on libc++.

When using the LLVM bootstrap to build LLVM and its libraries, overriding just LLVM to disable tests during the first build doesn’t seem to work. This stage name should remain stable, so check for it and disable tests when building in it.

Using the bootstrap stdenv avoids an infinite recursion from xcbuild depending on ncurses depending on xcrun from xcbuild, which is propagated by the non-bootstrap stdenv.

xarMinimal allows e2fsprogs to build fuse2fs again on Darwin. See NixOS#339412.

Take advantage of the new Darwin SDKs to dynamically determine SDK information such as path, version, and binaries (via `xcrun --find`). This is accomplished by relying on the existance of `DEVELOPER_DIR`, which the SDK will set up in nixpkgs.

xcbuild determines the SDK dynamically, so overriding the `sdkVer` no longer works. If packages want to change the SDK, they need to add one of the SDK packages to their inputs.

Enabling this by default requires additional clean that will be done after the Darwin refactor has been merged.

macOS ships with several stubs in `/usr/bin` that invoke `xcrun` to run the tools from the active SDK. When `/usr/bin` is in `PATH`, this will cause `xcrun` from xcbuild to invoke itself over and over. Filtering `/usr/bin` from `xcrun`’s search path prevents this from happening.

xcbuild is archived and will never be updated, so bump it to the last commit in the repo.

Using the bootstrap stdenv avoids an infinite recursion from xcbuild depending on zlib depending on xcrun from xcbuild, which is propagated by the non-bootstrap stdenv.

While it would be nice if this could be split, there are too many changes as part of the cleanup and improvements, including: - Refactoring all propagated packages into functions that can be used to ensure that packages are propagated only at the expected stages; - Using a sanity-checking merge function to ensure that packages are only propagated by one of the above functions; - Reducing the number of Python builds during the bootstrap to one; - Removing the extra sysctl stage; - Using the LLVM bootstrap to build LLVM, clang, libc++, etc; - Propagating llvmPackages_<version> in the final stdenv, so that packages needing that version specifically don’t have to rebuild it; - Bootstrapping with the new Darwin SDK; and - Reducing the overall number of paths build during a bootstrap by ~33%.

Using xarMinimal avoids pulling e2fsprogs and (eventually) macfuse-stubs into the Darwin stdenv bootstrap.

Static can rely on the standard cross-SDK because it will be a static SDK when building for static targets.

The patch adding support for `__ENVIRONMENT_OS_VERSION_MIN_REQUIRED__` to clang has been backported to all versions of clang in nixpkgs (except for clang 12), allowing this workaround to be dropped.

Use portable bmake instead of Apple’s port of BSD Make.

It’s not necessary with the new SDK.

Many packages need only libtool from cctools, which is different from GNU libtool (commonly used with other autotools), so it can’t be provided by default with the Darwin bintools. Providing it as a separate output allows packages to use cctools’s libtool without pulling other tools they may not want.

The SDK library path is normally add by the ld wrapper, but ld64 is being used unwrapped in this check phase. Ensure it can find the requires SDK stubs by passing them via the clang wrapper.

libunwind is included in the SDK, so it’s no longer necessary to include it as an explicit dependency.

Most Darwin source releases vendor private headers in a separate derivation and add them to `NIX_CFLAGS_COMPILE` (to avoid propagating them in static builds). Adopt this pattern for consistency with the other source releases.

Darwin’s libc is a stub that does nothing but avoid unwanted compiler and linker warnings. The actual libc is located dynamically based on the SDK. To support cross-compilation, the SDK is located based on platform: - `DEVELOPER_DIR_FOR_BUILD` - build platform; - `DEVELOPER_DIR` - host platform; and - `DEVELOPER_DIR_FOR_TARGET` - target platform.

libunwind is unnecessary on Darwin because it is included in the SDK, but setting libunwind to `null` would cause evaluation errors. Additionally, some packages expect to locate libunwind via pkg-config. Set the top-level libunwind to the Darwin compatibility package, so those packages can find it and use it.

`${stdenv.cc.libc}/lib/libSystem.B.dylib` does not exist and not existed for as long as I have used nixpkgs. Since `/usr/lib/libSystem.B.dylib` is already linked via text-based stubs, continue relying on those.

Change the adapter to add the requested SDK to the target’s build inputs. Note that only the “11.0” (i.e., 11.3) and 12.3 SDKs are supported. This adapter is retained for compatibility. The preferred way to override the SDK is to add it to the appropriate inputs directly.

configd is an alias for the SystemConfiguration framework, which is now always part of the SDK. Removing this parameter because it effectively does nothing now, which could be misleading to users.

The bootstrap tools linker sometimes crashes when trying to link the sqlite3 tests, which causes the bootstrap Python not to have the sqlite3 module. This causes the freezegun module to fail to build later in the bootstrap. Using the 11.0 SDK fixes the problem. Upstream Python supports building with a newer SDK and back-deploying, so this change should not negatively affect users on pre-11.0 releases.

These are a different kind of stubs that do nothing except tell you to check the documentation on how to use the new Darwin SDK instead.

Darwin uses the system unwinder, which is based on the LLVM unwinder. While it’s mostly API-compatible with libunwind, some packages expect to find it using `pkg-config`. This package provides a compatible file to allow those packages to use the system unwinder on Darwin.

apple-oss-distributions/adv_cmds@adv_cmds-163...adv_cmds-231

apple-oss-distributions/AvailabilityVersions@AvailabilityVersions-140.1...AvailabilityVersions-143.3

apple-oss-distributions/basic_cmds@basic_cmds-55...basic_cmds-70

apple-oss-distributions/bootstrap_cmds@bootstrap_cmds-121...bootstrap_cmds-136

apple-oss-distributions/copyfile@copyfile-138...copyfile-213

apple-oss-distributions/developer_cmds@developer_cmds-62...developer_cmds-79

apple-oss-distributions/diskdev_cmds@diskdev_cmds-593...diskdev_cmds-735

apple-oss-distributions/file_cmds@file_cmds-264.1.1...file_cmds-448.0.3

apple-oss-distributions/ICU@ICU-66108...ICU-74221

apple-oss-distributions/libiconv@libiconv-99...libiconv-107

apple-oss-distributions/libresolv@libresolv-64...libresolv-83

apple-oss-distributions/libutil@libutil-47.30.1...libutil-72

apple-oss-distributions/network_cmds@network_cmds-606.40.2...network_cmds-696

apple-oss-distributions/PowerManagement@PowerManagement-572.50.1...PowerManagement-1740.0.7

apple-oss-distributions/removefile@removefile-45...removefile-75

apple-oss-distributions/shell_cmds@shell_cmds-187...shell_cmds-319.0.1

apple-oss-distributions/system_cmds@system_cmds-970.0.4...system_cmds-1012

apple-oss-distributions/text_cmds@text_cmds-99...text_cmds-190.0.1

apple-oss-distributions/top@top-108...top-139

file_cmds 352.40.6 added a C-based implementation of xattr, which is the version of xattr used on recent releases of macOS. Align nixpkgs with what macOS is shipping, which has also the benefit of allowing xattr to be cross-compiled (previously precluded due to Python limitations).

apple-oss-distributions/xar@xar-498...xar-501

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

darwin: change the SDK pattern, update the SDKs, and update source releases #346043

darwin: change the SDK pattern, update the SDKs, and update source releases #346043

Commits on Oct 10, 2024