XcodeGen/xcconfig migration

[armcknight]

This discussion seeks feedback on moving to managing our Xcode projects and configurations in an XcodeGen spec and xcconfigs. It also serves as a memory aid for my own purposes as I have been working on this on a backburner for a while, as well as general tracking of tasks and visibility for open-source stakeholders.

Draft PR

Motivation

Removing the Xcode project and workspace from Git tracking and instead tracking an XcodeGen yml spec is easier to work on as a team. We have a large collection of separate Xcode projects which makes managing all the settings, targets and schemes unwieldy (11 projects and over 30 targets). Also, tracking Xcode project files in Git is a pain: merge conflicts can be tough to resolve (and automatic resolution sometimes lets spurious things creep in that don't actually break anything, but don't do anything either), and if you've modified the project then changing branches usually causes Xcode to ask about whether to resave/reload files or breaks the UI completely until you close/reopen.

Along with that, it's much cleaner to keep settings in xcconfig files vs Xcode projects, as you can deduplicate lots of things, and it makes it clearer what setting affects what targets.

And then, since the xcodeproj isn't in git, we wouldn't need to do what some other teams do by adding a Danger check to make sure nobody's committing settings to the xcodeproj instead of in xcconfigs.

Finally, going through the project and settings have teased out a few issues to fix.

TODOs

• extract all build settings
  • extract baseline settings before replacing with xcconfigs for validation after replacement
    • extract settings per SDK for Sentry since it compiles against iOS/macOS/watchOS/tvOS. they each output different settings
    • get the new PerformanceBenchmarks UI test target, with associated iOS-Swift-Benchmarking scheme, after merging ci: benchmarking updates #1926
  • extract xcconfigs and set in the projects
  • verify 100% parity between showBuildSettings output before/after
  • make sure it still works with SPM
• write xcodegen yml spec
  • capture targets
  • capture build phases
  • set xcconfigs
• add call to xcodegen in make init
• github actions updates
  • add triggers to workflows to run on changes to xcconfigs and the xcodegen project spec
  • add steps to workflows to generate the project

Extract all settings into xcconfigs

1. We have an xcworkspace with many projects–find them all

   ❯ find . -name "*.xcodeproj" -type d
   ./Sentry.xcodeproj
   ./Samples/Carthage-Validation/Framework/Framework.xcodeproj
   ./Samples/Carthage-Validation/XCFramework/XCFramework.xcodeproj
   ./Samples/watchOS-Swift/watchOS-Swift.xcodeproj
   ./Samples/iOS-SwiftUI/iOS-SwiftUI.xcodeproj
   ./Samples/tvOS-Swift/tvOS-Swift.xcodeproj
   ./Samples/iOS-Swift/iOS-Swift.xcodeproj
   ./Samples/TrendingMovies/TrendingMovies.xcodeproj
   ./Samples/iOS15-SwiftUI/iOS15-SwiftUI.xcodeproj
   ./Samples/macOS-Swift/macOS-Swift.xcodeproj
   ./Samples/iOS-ObjectiveC/iOS-ObjectiveC.xcodeproj
   # ... plus a few Carthage dependencies we don't care about ...

2. Ensured there was a scheme for every target in every project; we only had app schemes but not test schemes as the app/framework schemes directly ran test targets

3. Capture all build settings by running //scripts/output-all-schemes-showbuildsettings.sh

4. Used BuildSettingsExtractor to output xcconfigs for all Xcode projects in sentry-cocoa

   • this is necessarily manual as it’s a macOS GUI app only
     • file issue requesting a CLI tool!

5. Set the corresponding xcconfigs for every project’s target configuration file

   • Sentry.xcodeproj had some basic ones set for Sentry and SentryTests targets; removed those old xcconfigs

6. Deleted all pbxproj build settings for every project/target

7. As in (3), again capture output of xcodebuild -showBuildSettings for every target of every project under each configuration (all had Debug/Release only, with the exception of Sentry.xcodeproj which also has Test/TestCI) into text files in an after/ directory, for comparison

   1. all the same scripts, just replace xcodebuild_showBuildSettings_before with xcodebuild_showBuildSettings_after

8. Verify parity of build settings by comparing before/ and after/ directories and finding no differences

   1. diff xcodebuild_showBuildSettings_before/ xcodebuild_showBuildSettings_after/ | sort | uniq -c shows all the actual differences (and how many files contained each), such as:

      16 <     CLANG_CXX_LANGUAGE_STANDARD = c++14
      16 <     CURRENT_PROJECT_VERSION = 7.22.0
      16 <     DYLIB_COMPATIBILITY_VERSION = 1
      16 <     DYLIB_CURRENT_VERSION = 1
       4 <     LD_RUNPATH_SEARCH_PATHS =  @executable_path/../Frameworks @loader_path/../Frameworks @executable_path/Frameworks @loader_path/Frameworks
      12 <     LD_RUNPATH_SEARCH_PATHS =  @executable_path/Frameworks @loader_path/Frameworks @executable_path/Frameworks @loader_path/Frameworks
      16 <     MODULEMAP_FILE = /Users/andrewmcknight/Code/organization/getsentry/repos/public/sentry-cocoa/Sources/Sentry/Sentry.modulemap
      16 <     SDKROOT__CARTHAGE_ = iphoneos
      16 <     SDKROOT__CARTHAGE_NO = iphoneos
      16 <     SDKROOT__CARTHAGE_YES = macosx
      16 <     SKIP_INSTALL = YES
      16 <     SUPPORTED_PLATFORMS = macosx iphoneos iphonesimulator watchos watchsimulator appletvos appletvsimulator
      16 <     SUPPORTS_MACCATALYST = YES
       4 <     SWIFT_COMPILATION_MODE = wholemodule
       4 <     SWIFT_OPTIMIZATION_LEVEL = -O
      16 <     TARGETED_DEVICE_FAMILY = 1,2,3,4
      16 <     VERSION_INFO_STRING = "@(#)PROGRAM:Sentry  PROJECT:Sentry-7.22.0"
      16 >     CLANG_CXX_LANGUAGE_STANDARD = gnu++0x
      16 >     LD_RUNPATH_SEARCH_PATHS =  @executable_path/Frameworks @loader_path/Frameworks
      16 >     SKIP_INSTALL = NO
       4 >     SUPPORTED_PLATFORMS = appletvos appletvsimulator
       4 >     SUPPORTED_PLATFORMS = iphonesimulator iphoneos
       4 >     SUPPORTED_PLATFORMS = macosx
       4 >     SUPPORTED_PLATFORMS = watchsimulator watchos
       4 >     SWIFT_OPTIMIZATION_LEVEL = -Owholemodule
      16 >     TARGETED_DEVICE_FAMILY = 1,2
      16 >     VERSION_INFO_STRING = "@(#)PROGRAM:Sentry  PROJECT:Sentry-"

9. mend any differences in subsequent commits

10. do another output to after-fixed/ which should be identical to before/

Capture workspace structure in XcodeGen spec

need to refactor xcconfigs to combine the previously separate xcodeproj’ project-level xcconfigs into one for the new single project

Irregularities fixed

• removed a few test targets that had no test cases:
  • iOS15-SwiftUITests
  • iOS15-SwiftUIUITests
  • ref: remove unused test targets #2069

- Removed a bunch of test resources that were unused: I wound up finding where these were used
- SentryFileIOTrackingIntegrationTests was in SentryTests and iOS-SwiftUITests; should not be in the latter, it’s not even a UI test Dhiogo pointed out why this is the case

• some targets were set to gnu c++ compilers; set them all to c++14: fix: standardize on C++14 #2068
• fix LD_RUNPATH_SEARCH_PATHS for macOS compilation of Sentry framework (was set to include the iOS runpaths)
• removed some outdated build settings
  • SWIFT_SWIFT3_OBJC_INFERENCE = Off
• unbundle the multiplatform framework target into separate targets for each platform
  • LD_DYLIB_INSTALL_NAME points to the correct macOS location now (@rpath/Sentry.framework/Versions/A/Sentry instead of @rpath/Sentry.framework/Sentry like for iOS etc)
  • don’t have to do the weird carthage workaround for SDKROOT
• several settings tried to use config conditions that were not allowed
• sentry supports back to tvos 9 but our sample tvos apps only went back to tvos 13
  • discovered but not fixed yet because it requires code changes in the sample app implementation
  • likewise with watchOS; we declare support back to 2, but our sample app only supports back to 6.2
• standardized the name of some UI test targets to clearly disambiguate what kind of tests they are (eg, pop quiz: if we have a target named iOS-SwiftUI, is iOS-SwiftUITests a unit test or ui test target? is iOS-SwiftUITests a UI test for iOS-Swift or a unit test for iOS-SwiftUI?)
• some headers were added to the codebase that weren't added to the framework target's headers phase: SentryCrashIsAppImage.h, SentryEvent+Private.h
• failure to build macOS-Swift:
  • error building CppWrapper.m, possibly because it was compiled with GNU extensions, but changing it from .m to .mm fixes the build error
  • error building Samples/macOS-Swift/macOS-Swift/ViewController.swift because of a wrongly declared selector (this was previously a warning that got upgraded to an error, because GCC_TREAT_WARNINGS_AS_ERROS was previously only set to YES for the Sentry framework targets, and it's now set to YES project-wide)
• removed several source files that existed in the codebase but were never compiled into a test target, and caused build failures when added by XcodeGen
  • TestSentryCrashWrapper.swift (appears to have been rewritten in objc)
  • SentryRateLimitsParserTests.m (appears to have been rewritten in swift)
  • Tests/SentryTests/Integrations/Performance/CoreData/TestModel.xcdatamodeld/TestModel.xcdatamodel/contents and since it had automatic code generation, the generated classes conflicted with others that were hand written and actually compiled into the tests
  • SentrySerializationTests.m (rewritten in swift)
  • SentryStacktraceEquality.swift (empty)
• iOS-Swift sets SUPPORTS_MACCATALYST to YES but doesn't build for it:
  /Users/andrewmcknight/Code/organization/getsentry/repos/public/sentry-cocoa-clone-always-master/Samples/iOS-Swift/iOS-Swift.xcodeproj Building for iOS, but the linked and embedded framework 'Sentry.framework' is building for Mac Catalyst. You may need to configure 'Sentry.framework' to build for iOS.
  • so far, got it building now, but fails at runtime due to error:
    dyld[21098]: Library not loaded: @rpath/Sentry.framework/Versions/A/Sentry Referenced from: /Users/andrewmcknight/Library/Developer/Xcode/DerivedData/Sentry-ewgvnkbxbplozgbzvtnbhxsvcpmm/Build/Products/Debug-maccatalyst/iOS-Swift.app/Contents/MacOS/iOS-Swift Reason: tried: '/Users/andrewmcknight/Library/Developer/Xcode/DerivedData/Sentry-ewgvnkbxbplozgbzvtnbhxsvcpmm/Build/Products/Debug-maccatalyst/Sentry.framework/Versions/A/Sentry' (code signature in <460E704C-3130-32B2-8415-9A4EA208213C> '/Users/andrewmcknight/Library/Developer/Xcode/DerivedData/Sentry-ewgvnkbxbplozgbzvtnbhxsvcpmm/Build/Products/Debug-maccatalyst/Sentry.framework/Versions/A/Sentry' not valid for use in process: Trying to load an unsigned library), '/Users/andrewmcknight/Library/Developer/Xcode/DerivedData/Sentry-ewgvnkbxbplozgbzvtnbhxsvcpmm/Build/Products/Debug-iphoneos/Sentry.framework/Versions/A/Sentry' (no such file), '/Users/andrewmcknight/Library/Developer/Xcode/DerivedData/Sentry-ewgvnkbxbplozgbzvtnbhxsvcpmm/Build/Products/Debug-maccatalyst/PackageFrameworks/Sentry.framework/Versions/A/Sentry' (no such file), '/Users/andrewmcknight/Library/Developer/Xcode/DerivedData/Sentry-ewgvnkbxbplozgbzvtnbhxsvcpmm/Build/Products/Debug-iphoneos/PackageFrameworks/Sentry.framework/Versions/A/Sentry' (no such file), '/usr/lib/swift/Sentry.framework/Versions/A/Sentry' (no such file), '/Users/andrewmcknight/Library/Developer/Xcode/DerivedData/Sentry-ewgvnkbxbplozgbzvtnbhxsvcpmm/Build/Products/Debug-maccatalyst/iOS-Swift.app/Contents/MacOS/Frameworks/Sentry.framework/Versions/A/Sentry' (no such file), '/Users/andrewmcknight/Library/Developer/Xcode/DerivedData/Sentry-ewgvnkbxbplozgbzvtnbhxsvcpmm/Build/Products/Debug-maccatalyst/iOS-Swift.app/Contents/MacOS/Frameworks/Sentry.framework/Versions/A/Sentry' (no such file), '/usr/lib/swift/Sentry.framework/Versions/A/Sentry' (no such file), '/Users/andrewmcknight/Library/Developer/Xcode/DerivedData/Sentry-ewgvnkbxbplozgbzvtnbhxsvcpmm/Build/Products/Debug-maccatalyst/iOS-Swift.app/Contents/MacOS/Frameworks/Sentry.framework/Versions/A/Sentry' (no such file), '/Users/andrewmcknight/Library/Developer/Xcode/DerivedData/Sentry-ewgvnkbxbplozgbzvtnbhxsvcpmm/Build/Products/Debug-maccatalyst/iOS-Swift.app/Contents/MacOS/Frameworks/Sentry.framework/Versions/A/Sentry' (no such file), '/System/Library/Frameworks/Sentry.framework/Versions/A/Sentry' (no such file)
  • looks like we need a test for this? we run SentryTests on Catalyst, but don't try to embed it into a test app to make sure that works; and it may not be working!
• fixed code sign settings so we don't have to make a bunch of changes to settings in the xcode ui every time we want to test on device
  • make automatic in the project, then set to match-provided stuff in fastfile when running in CI
• we include some headers from SentryTests into the framework:
  • //Tests/SentryTests/Helper/SentryTestObjCRuntimeWrapper.h
  • //Tests/SentryTests/TestUtils/SentryClassRegistrator.h

Other improvements

• adds the macOS-SPM-CommandLine sample app and test-server to the new project
• abandons the multiplatform target approach to instead have a separate target per platform for the framework; we no longer need these carthage workarounds or the make build-for-watchos XCFramework build step

[brustolin]

A few points to consider

• Xcode asking for reload after changing branches will continue to happen.
• We lose the ability to configure any part of the project using the IDE. We have to learn a new way to configure it and every new contractor, employee or open source contributor will needs to know how to use xcodegen.
• If Apple makes change in the xCodeProj format in a new Xcode release, we need to wait for xCodeGen to update.
• If xcodegen stops being maintained we have to roll back the project.
• In one and a half year working in this project I haven't seen any problem regarding xcodeproj merge. This is the biggest problem xCodeGen will solve, but this hasn't being a problem at all.

In summary, we gonna have a big change in paradigm in our project that does not really solves a problem.

[mitsuhiko]

I am not sure if we tried this at Sentry but I only made bad experiences with trying to generate xcode files. Be it cmake, premake or whatever else I used before. Eventually you run into the tool not keeping up with Apple's changes so I'm somewhat on the side of trying to stay away from project file generation just based on that.

[armcknight]

I'm going to start a separate thread per point in here to keep this one from going long.

[armcknight]

Xcode asking for reload after changing branches will continue to happen.
This is only if the project file has changed, not every branch change.

This change still lessens the overall pain level, because it will prevent Xcode from throwing up dialogs every time the tracked project changes, like during a rebase, instead of every time the generated project changes, which is controlled by us in our own private workflows.

[kevinrenskers]

In 6 weeks I've only had to reload Xcode when I got a merge conflict in the project file, 2 or 3 times. Other than that, switching branches and all that, I haven't seen any issues with Xcode.

[philipphofmann]

I agree with what you stated here, @kevinrenskers. I don't experience any issues with it how it is now.

[armcknight]

We lose the ability to configure any part of the project using the IDE. We have to learn a new way to configure it and every new contractor, employee or open source contributor will needs to know how to use xcodegen.

This is a well-known tool in the iOS community. And it's a simple spec language. Please have a read through the project.yml and point out anything you don't understand or think would be confusing about it.

[kevinrenskers]

You kinda skipped over the "we lose the ability to configure any part of the project using the IDE" part though.

[kevinrenskers]

This is a well-known tool in the iOS community.

I've been doing iOS since 2010 and I've never used it, never seen a client who uses it. I think the argument that @brustolin made is valid: "We have to learn a new way to configure it and every new contractor, employee or open source contributor will needs to know how to use xcodegen".

[armcknight]

If Apple makes change in the xCodeProj format in a new Xcode release, we need to wait for xCodeGen to update. If xcodegen stops being maintained we have to roll back the project.

Eventually you run into the tool not keeping up with Apple's changes

If this happens, we can do 2 things:

1. Fork XcodeGen
2. Check in the project generated from project.yml on master. This is not the same as a "rollback" though, there would be no manual editing of the project file, it would be good to go as-is.

[kevinrenskers]

As @mitsuhiko said:

Eventually you run into the tool not keeping up with Apple's changes so I'm somewhat on the side of trying to stay away from project file generation just based on that.

And I 100% agree with that. What if Xcode 15 drastically changes the project format and XcodeGen takes a few months to support it?

[armcknight]

In one and a half year working in this project I haven't seen any problem regarding xcodeproj merge. This is the biggest problem xCodeGen will solve, but this hasn't being a problem at all.

Within one month of working on it I had annoyances as well as experiencing merge conflict issues. Just the constant switching of the watchOS app's xcscheme is annoying to no end.

[kevinrenskers]

I do agree with that. I've worked on this project for about 5 or 6 weeks now and have had 2 or 3 project file merge conflicts. But then again they were extremely simple to solve as I could just accept both changes. The watchOS app scheme annoyance.. I think we simply need to commit the change that keeps constantly popping up? I've definitely noticed the same though.

[philipphofmann]

Yes, I agree. It can be a bit annoying sometimes, but it's not up to a point where I would take action to improve it. xCodeGen will for sure also bring its annoyances. Remember that we all have to learn something new, and we will ping you a lot in the beginning @armcknight. But maybe it's way better, and I just don't want it because I don't know it 😁.

[armcknight]

For me a huge huge downside (a dealbreaker really) is that we now have one huge list of files in Xcode. We could create folders on disk and organize them that way, but then we'll no doubt run into problems with the header files which need to be in specific public or include folders, so would no longer appear alongside the .m file.

The public/include/etc header files actually don't need to be in specific folders like we currently have them. And if we're going to change the whole organization of files on disk, since that's where the real flatness currently is, we might as well do away with this idea as well (which I'm happy to do!)

[armcknight]

I only hesitated to do the folder organization due to git history issues but I think now actually most tooling correctly traverses renames.

[armcknight]

This will also provide an opportunity to simplify the way we currently organize our headers.

From CONTRIBUTING.md:

To make a header...Move it into the folder Public

XcodeGen defaults to making headers public (this is sadly not the same as the Xcode default for a new .h/.m pair created, which defaults to project; see yonaskolb/XcodeGen#1244).

We consistently follow this rule in the current project. However, we have a //Sources/Sentry/include that has some headers included with project visibility and others with private. I think we can remove //Sources/Sentry/include and put those headers next to their implementations.

I will also look at symlinking headers into dedicated public/project/private folders to see if we can easily organize headers like that for deployment, while keeping all actual headers alongside their implementations.

[armcknight]

I'm going to shelve the xcodegen refactor for now and focus on fixing things in the "fixed" section of this discussion within the context of our current project structure.