Error waiting for board discovery on Arduino IDE 2.3.3 on OSX ARM64

[tiagosr]

When the Arduino IDE auto-updated from 2.3.2 to 2.3.3 with arduino-pico installed on my M1 Macbook Pro running OSX 12.6.1, all of the managers, menus and buttons in the IDE lost functionality. Reverting to 2.3.2 returned the IDE to a working state, and attempting a new update triggered the error again. Removing the ~/Library/Arduino15/packages/rp2040 folder returns 2.3.3 to a working state, but reinstalling arduino-pico caused all the IDE background servers to stop responding.

I noticed these lines in the stdout/stderr output of Arduino IDE:

2024-10-04T20:45:26.042Z root INFO Deploy plugins list: 16.2 ms [Finished 0.911 s after backend start]
2024-10-04T20:45:26.061Z discovery-log INFO start resolved watching
2024-10-04T20:45:26.066Z daemon INFO time="2024-10-04T22:45:26+02:00" level=error msg="Waiting discovery process termination: signal: abort trap" discovery=rp2040
2024-10-04T20:45:26.069Z daemon INFO panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x8 pc=0x100e4b374
2024-10-04T20:45:26.069Z daemon INFO ]

goroutine 54 [running]:
github.com/arduino/pluggable-discovery-protocol-handler/v2.(*Client).Run(
2024-10-04T20:45:26.069Z daemon INFO 0x140001623f0)
	/go/pkg/mod/github.com/arduino/pluggable-discovery-protocol-handler/v2@v2.2.0/client.go:281 +0x124
github.com/arduino/arduino-cli/internal/arduino/discovery/discoverymanager.(*DiscoveryManager).startDiscovery(0x14000236550, 0x140001623f0)
	/home/build/internal/arduino/discovery/discoverymanager/discoverymanager.go:203 +0x68
github.com/arduino/arduino-cli/internal/arduino/discovery/discoverymanager.(*DiscoveryManager).Start.func2(0x0?)
	/home/build/internal/arduino/discovery/discoverymanager/discoverymanager.go:105 +0x40
2024-10-04T20:45:26.069Z daemon INFO created by github.com/arduino/arduino-cli/internal/arduino/discovery/discoverymanager.(*DiscoveryManager).Start in goroutine 50
	/home/build/internal/arduino/discovery/discoverymanager/discoverymanager.go:104 +0x1a4
2024-10-04T20:45:26.074Z daemon INFO Daemon exited with exit code: 2.
2024-10-04T20:45:26.074Z discovery-log INFO error received
2024-10-04T20:45:26.074Z discovery-log INFO cancel error received!
2024-10-04T20:45:26.074Z discovery-log INFO received end

[earlephilhower]

Moved to a discussion because this looks like either an IDE-specific problem or a specific computer issue. If we start getting tons of reports about this we can look into it, but the last touch to pluggable_discovery.py was 2 years ago and it's really basic (all this module does is print stuff to stdout when it sees a UF2 board appear, OTW just stay silent).

The fact that you're showing the IDE crashing 2024-10-04T20:45:26.069Z daemon INFO panic: runtime error: invalid memory address or nil pointer dereference (this core's discovery runs in an entirely different process) makes me think there are system or installation problems. Can you blow away all IDE installs and your ~/.arduino15 folder where installed packages are kept, and reinstall everything from scratch?

[tiagosr]

Digging a little deeper into it, attempting to run ~/Library/Arduino15/packages/rp2040/tools/pqt-python3/1.0.1-base-3a57aed-1/python3 outputs the following error:

../pqt-python3/1.0.1-base-3a57aed-1 ❯❯❯ ./python3 
dyld[46893]: Symbol not found: (_mkfifoat)
  Referenced from: '/Users/tiago/Library/Arduino15/packages/rp2040/tools/pqt-python3/1.0.1-base-3a57aed-1/python3'
  Expected in: '/usr/lib/libSystem.B.dylib'
zsh: abort      ./python3

Would this possibly be related to the crash?

[earlephilhower]

You should be fine with your changes. The .ini file is modified slightly from the git version on release to work with the IDE tools, so there will be minor diffs like that. The main thing is to get rid of references to the discovery Python script, which you've done.

[earlephilhower]

Would this possibly be related to the crash?

The runtime error you've pasted would cause python to simply not run at all. You would never be able to actually link any sketches, either, if that was the case because simplesub.py needs python3 in order to make the custom per-app linker file. So, if things are building and uploading, I wouldn't bother too much.

If nothing is working, did you upgrade the OS or do a system update between the last successful run and now? Python3 has also been untouched on ARM Mac for 9 months (looking at the git history in pico-quick-toolchain). Apple could have decided to break "old" apps by removing a syscall they supported in MacOS 14 (XCode 14.3.1)...if so, every upgraded ARM Mac should fail in the same way and a new build would be needed...breaking compatibility for some number of older Macs. 😢

[tiagosr]

Thanks, switching to using the {runtime.platform.path} based python executables did the trick for things to work in 2.3.3, including device discovery.

The runtime error you've pasted would cause python to simply not run at all. You would never be able to actually link any sketches, either,

Yes, it was the case indeed - but even more insidiously the entire IDE became mostly unresponsive, I couldn't even trigger a compile or even load a sketch, as the sketch manager showed up empty. That crash caused the background servers to just give up responding to anything else, which might be due to some parameter checks being missing in the Go side of things in the IDE.

As far as updates go, there were some updates recently but I am still on MacOS 12 in this machine - it could still be that they updated python to remove the syscall, but the same code and packages were also still working when reverting to 2.3.2 in the same machine. There may be something else in the 2.3.3 environment that caused pico-quick-toolchain's version of python to stop working, bringing down the IDE's servers along with it.

[earlephilhower]

The toolchain runs under MacOS 12 for x86-64, we do it under CI:

arduino-pico/.github/workflows/pull-request.yml

Lines 194 to 199 in 2324d52

	# Single build under macOS to ensure the Mac toolchain is good.
	build-mac:
	name: Mac
	strategy:
	matrix:
	os: [macOS-12, macOS-14]

... is there some command to dump detailed update levels to compare CI and your box?

What exactly did you edit related to runtime.platform.path? I just d/l'd the release ZIP and it calls {runtime.tools.xxxx} for everything (GCC, picotool, etc.), as expected.

[sdc01]

Found a similar issue on Mac OS 12.7.6 running on Apple M2 Silicon. Sounds like an Apple OS issue but if there is a work around it would help everyone on Macs.

When compiling code for Cytron Maker Uno RP2040 and other RP2040 devices this message shows up:

dyld[4183]: Symbol not found: (_mkfifoat)
Referenced from: '/Users/steve/Library/Arduino15/packages/rp2040/tools/pqt-python3/1.0.1-base-3a57aed-1/python3'
Expected in: '/usr/lib/libSystem.B.dylib'

and it ends with

signal: abort trap
Compilation error: signal: abort trap

I found this info about the dylib file in the Apple Developer Forum.

Problems like this typically happen when folks try to link against the dynamic libraries built in to the system rather than the stub libraries in the macOS SDK. I talk about this concept in general in An Apple Library Primer, and if you search that post for “stub library” you’ll find a link to another post where I explain it in more detail. In short:

The /usr/lib/libSystem.B.dylib dynamic library no longer exists on disk because it was rolled into the dynamic linker shared cache.

If you set up your build correctly, you don’t need this file because the linker will use the usr/lib/libSystem.B.tbd stub library from the macOS SDK.

As to how you can fix this, it’s hard to say because I don’t have any experience with this specific third-party library. My general advice is that you escalate questions like this to the library developer. However, it sounds like you’ve done that already, with no success. Your only remaining option is to debug the library’s build system yourself. You need to work out why it’s referencing /usr/lib/libSystem.B.dylib directly, rather than the usr/lib/libSystem.B.tbd stub library in the macOS SDK.

[earlephilhower]

I built python3 in a GitHub CI instance since I don't run any Macs here. It's all scripted in CI: https://github.com/earlephilhower/pico-quick-toolchain/blob/master/.github/workflows/build-macarm-python.yml

It's almost trivial and basically the same ./configure && make that you'd run on anything else with a slight modification to make Python3 a portable installation and not require system-wide installation.

I think it's just that there was a call in the XCode 14.3.1 stub libraries or the GH Mac VM/docker image that's not present in certain MacOS versions. (FWIW I tried to be as trailing in Xcode version as possible for the build because I know Apple drops system support over time...)