Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python312Packages.kaleido: init at 0.2.1 #339136

Merged
merged 5 commits into from
Sep 5, 2024

Conversation

Pandapip1
Copy link
Contributor

@Pandapip1 Pandapip1 commented Sep 3, 2024

Description of changes

I'm sorry, I literally couldn't figure out how to do a source build. If someone would like to figure it out for themselves, uh, good luck!

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

pkgs/development/python-modules/kaleido/default.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/kaleido/default.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/kaleido/default.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/kaleido/default.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/kaleido/default.nix Outdated Show resolved Hide resolved
"aarch64-linux"
"aaarch64-darwin"
];
sourceProvenance = with lib.sourceTypes; [ binaryNativeCode ]; # Trust me, I'm not happy. But after literal hours of trying to reverse-engineer their build system and getting nowhere, I'll use the stupid binaries >:(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the third attempt, as far as I know, but no one has been able to build it from the source.😢

@Pandapip1 Pandapip1 force-pushed the init-python3packages-kaleido branch 4 times, most recently from 4ce530b to f30a433 Compare September 3, 2024 04:48
@Pandapip1 Pandapip1 changed the title kaleido: init at 0.2.1.post1 kaleido: init at 0.2.1 Sep 3, 2024
@Pandapip1 Pandapip1 force-pushed the init-python3packages-kaleido branch 3 times, most recently from 9add00d to a411299 Compare September 3, 2024 05:12
@eclairevoyant eclairevoyant changed the title kaleido: init at 0.2.1 python312Packages.kaleido: init at 0.2.1 Sep 3, 2024
@eclairevoyant
Copy link
Contributor

Also please use the full attrname in commits so that borgo knows what to build (i.e. python312Packages.kaleido and same with plotly)

@Pandapip1
Copy link
Contributor Author

Just noticed there weren't any maintainers for plotly, so this PR also has me adopting it.

@Pandapip1 Pandapip1 added the 0.kind: package adoption Requests for adopting packages that have no maintainers label Sep 3, 2024
@eclairevoyant
Copy link
Contributor

eclairevoyant commented Sep 3, 2024

please use python312Packages, not python3Packages, in the commit message.

@natsukium
Copy link
Member

@ofborg build python312Packages.kaleido python311Packages.kaleido

@eclairevoyant
Copy link
Contributor

I think I figured out what you meant. Does ofborg not recognize the python3Packages.xxx format?

Right, hence the note here to explicitly specify the version:

* Commit names of Python libraries should reflect that they are Python
libraries, so write for example `python311Packages.numpy: 1.11 -> 1.12`.
It is highly recommended to specify the current default version to enable
automatic build by ofborg.

@Pandapip1
Copy link
Contributor Author

Okay, that's not particularly clear. I'll open a PR to rephrase that.

@natsukium
Copy link
Member

Tests fail on darwin with the following log.
*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Critical error: required built-in appearance SystemAppearance not found'

I suspect CoreFoundation, AppKit, or the matplotlib environment variable MPLBACKEND, but I can't fix it.
Maybe someone at @NixOS/darwin-maintainers knows how to make it work.

kaleido-tests-aarch64-darwin> building '/nix/store/r6dib0z8rc7pbjhfqrh184a6vh9k9a8h-kaleido-tests.drv'
kaleido-tests-aarch64-darwin> Traceback (most recent call last):
kaleido-tests-aarch64-darwin>   File "/nix/store/i36nyn014clgrvrgmv60vg58wl0ifmvn-tests.py", line 11, in <module>
kaleido-tests-aarch64-darwin>     fig.write_image(outfile, engine="kaleido")
kaleido-tests-aarch64-darwin>   File "/nix/store/k4hlqc8dcdw3890fvghzl3sppwfcs78p-python3.12-plotly-5.23.0/lib/python3.12/site-packages/plotly/basedatatypes.py", line 3835, in write_image
kaleido-tests-aarch64-darwin>     return pio.write_image(self, *args, **kwargs)
kaleido-tests-aarch64-darwin>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kaleido-tests-aarch64-darwin>   File "/nix/store/k4hlqc8dcdw3890fvghzl3sppwfcs78p-python3.12-plotly-5.23.0/lib/python3.12/site-packages/plotly/io/_kaleido.py", line 266, in write_image
kaleido-tests-aarch64-darwin>     img_data = to_image(
kaleido-tests-aarch64-darwin>                ^^^^^^^^^
kaleido-tests-aarch64-darwin>   File "/nix/store/k4hlqc8dcdw3890fvghzl3sppwfcs78p-python3.12-plotly-5.23.0/lib/python3.12/site-packages/plotly/io/_kaleido.py", line 143, in to_image
kaleido-tests-aarch64-darwin>     img_bytes = scope.transform(
kaleido-tests-aarch64-darwin>                 ^^^^^^^^^^^^^^^^
kaleido-tests-aarch64-darwin>   File "/nix/store/y1fsgdcc12c2jyb1pal6ds0icrz7x8x2-python3.12-kaleido-0.2.1/lib/python3.12/site-packages/kaleido/scopes/plotly.py", line 153, in transform
kaleido-tests-aarch64-darwin>     response = self._perform_transform(
kaleido-tests-aarch64-darwin>                ^^^^^^^^^^^^^^^^^^^^^^^^
kaleido-tests-aarch64-darwin>   File "/nix/store/y1fsgdcc12c2jyb1pal6ds0icrz7x8x2-python3.12-kaleido-0.2.1/lib/python3.12/site-packages/kaleido/scopes/base.py", line 293, in _perform_transform
kaleido-tests-aarch64-darwin>     self._ensure_kaleido()
kaleido-tests-aarch64-darwin>   File "/nix/store/y1fsgdcc12c2jyb1pal6ds0icrz7x8x2-python3.12-kaleido-0.2.1/lib/python3.12/site-packages/kaleido/scopes/base.py", line 198, in _ensure_kaleido
kaleido-tests-aarch64-darwin>     raise ValueError(message)
kaleido-tests-aarch64-darwin> ValueError: Failed to start Kaleido subprocess. Error stream:
kaleido-tests-aarch64-darwin> 
kaleido-tests-aarch64-darwin> [0905/000022.334097:WARNING:resource_bundle.cc(431)] locale_file_path.empty() for locale en-US
kaleido-tests-aarch64-darwin> *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Critical error: required built-in appearance SystemAppearance not found'
kaleido-tests-aarch64-darwin> *** First throw call stack:
kaleido-tests-aarch64-darwin> (
kaleido-tests-aarch64-darwin> 	0   CoreFoundation                      0x0000000186e662ec __exceptionPreprocess + 176
kaleido-tests-aarch64-darwin> 	1   libobjc.A.dylib                     0x000000018694a158 objc_exception_throw + 60
kaleido-tests-aarch64-darwin> 	2   CoreFoundation                      0x0000000186e661dc +[NSException exceptionWithName:reason:userInfo:] + 0
kaleido-tests-aarch64-darwin> 	3   AppKit                              0x000000018a61e214 __33+[NSAppearance _initializeCoreUI]_block_invoke + 88
kaleido-tests-aarch64-darwin> 	4   libdispatch.dylib                   0x0000000186b613e8 _dispatch_client_callout + 20
kaleido-tests-aarch64-darwin> 	5   libdispatch.dylib                   0x0000000186b62c68 _dispatch_once_callout + 32
kaleido-tests-aarch64-darwin> 	6   AppKit                              0x000000018a63e5cc +[NSAppearance _aquaAppearance] + 60
kaleido-tests-aarch64-darwin> 	7   AppKit                              0x000000018a61d98c +[NSAppearance appearanceNamed:] + 32
kaleido-tests-aarch64-darwin> 	8   AppKit                              0x000000018a61d124 -[NSSystemAppearanceProxy init] + 124
kaleido-tests-aarch64-darwin> 	9   AppKit                              0x000000018a61d098 __38+[NSSystemAppearanceProxy systemProxy]_block_invoke + 24
kaleido-tests-aarch64-darwin> 	10  libdispatch.dylib                   0x0000000186b613e8 _dispatch_client_callout + 20
kaleido-tests-aarch64-darwin> 	11  libdispatch.dylib                   0x0000000186b62c68 _dispatch_once_callout + 32
kaleido-tests-aarch64-darwin> 	12  AppKit                              0x000000018a61d07c +[NSSystemAppearanceProxy systemProxy] + 64
kaleido-tests-aarch64-darwin> 	13  AppKit                              0x000000018a61d008 -[NSApplication(NSApplicationAppearance_Internal) _registerForAppearanceNotifications] + 32
kaleido-tests-aarch64-darwin> 	14  AppKit                              0x000000018a61acec -[NSApplication init] + 904
kaleido-tests-aarch64-darwin> 	15  AppKit                              0x000000018a61a798 +[NSApplication sharedApplication] + 128
kaleido-tests-aarch64-darwin> 	16  kaleido                             0x000000010908fb74 _ZN8headless27HeadlessContentMainDelegate24PreCreateMainMessageLoopEv + 36
kaleido-tests-aarch64-darwin> 	17  kaleido                             0x0000000106fb9748 _ZN7content21ContentMainRunnerImpl17RunServiceManagerERNS_18MainFunctionParamsEb + 188
kaleido-tests-aarch64-darwin> 	18  kaleido                             0x0000000106fb9660 _ZN7content21ContentMainRunnerImpl3RunEb + 304
kaleido-tests-aarch64-darwin> 	19  kaleido                             0x0000000106fb7e40 _ZN7content17RunContentProcessERKNS_17ContentMainParamsEPNS_17ContentMainRunnerE + 552
kaleido-tests-aarch64-darwin> 	20  kaleido                             0x0000000106fb8520 _ZN7content11ContentMainERKNS_17ContentMainParamsE + 48
kaleido-tests-aarch64-darwin> 	21  kaleido                             0x0000000107041240 _ZN8headless12_GLOBAL__N_114RunContentMainENS_15HeadlessBrowser7OptionsEN4base12OnceCallbackIFvPS1_EEE + 260
kaleido-tests-aarch64-darwin> 	22  kaleido                             0x0000000107041104 _ZN8headless19HeadlessBrowserMainENS_15HeadlessBrowser7OptionsEN4base12OnceCallbackIFvPS0_EEE + 56
kaleido-tests-aarch64-darwin> 	23  kaleido                             0x000000010444b8e4 main + 180
kaleido-tests-aarch64-darwin> 	24  dyld                                0x0000000186987154 start + 2476
kaleido-tests-aarch64-darwin> )
kaleido-tests-aarch64-darwin> /nix/store/y1fsgdcc12c2jyb1pal6ds0icrz7x8x2-python3.12-kaleido-0.2.1/lib/python3.12/site-packages/kaleido/executable/kaleido: line 5:  3017 Trace/BPT trap: 5       ./bin/kaleido $@

@Pandapip1
Copy link
Contributor Author

@natsukium in the meantime, I've marked the package as broken on darwin. Does that address that concern for the time being?

@natsukium
Copy link
Member

I think it looks good. The darwin fix can be done in another PR.

@natsukium
Copy link
Member

cc @GaetanLepage, @Sigmanificient if you're interested.

@Pandapip1
Copy link
Contributor Author

Pandapip1 commented Sep 5, 2024

Result of nixpkgs-review pr 339136 run on x86_64-linux 1

10 packages marked as broken and skipped:
  • picosnitch
  • picosnitch.dist
  • python311Packages.pymatgen
  • python311Packages.pymatgen.dist
  • python311Packages.sumo
  • python311Packages.sumo.dist
  • python312Packages.pymatgen
  • python312Packages.pymatgen.dist
  • python312Packages.sumo
  • python312Packages.sumo.dist
2 packages failed to build:
  • theharvester
  • theharvester.dist
120 packages built:
  • apkleaks
  • apkleaks.dist
  • checkov
  • checkov.dist
  • cve-bin-tool
  • cve-bin-tool.dist
  • das
  • das.dist
  • deeptools
  • deeptools.dist
  • hp2p
  • jadx
  • python311Packages.ax-platform
  • python311Packages.ax-platform.dist
  • python311Packages.catboost
  • python311Packages.catboost.dist
  • python311Packages.chart-studio
  • python311Packages.chart-studio.dist
  • python311Packages.compressai
  • python311Packages.compressai.dist
  • python311Packages.cufflinks
  • python311Packages.cufflinks.dist
  • python311Packages.dash
  • python311Packages.dash-bootstrap-components
  • python311Packages.dash-bootstrap-components.dist
  • python311Packages.dash.dist
  • python311Packages.experiment-utilities
  • python311Packages.experiment-utilities.dist
  • python311Packages.explorerscript
  • python311Packages.explorerscript.dist
  • python311Packages.igraph
  • python311Packages.igraph.dist
  • python311Packages.kaleido
  • python311Packages.kaleido.dist
  • python311Packages.kmapper
  • python311Packages.kmapper.dist
  • python311Packages.leidenalg
  • python311Packages.leidenalg.dist
  • python311Packages.meshtastic
  • python311Packages.meshtastic.dist
  • python311Packages.optuna
  • python311Packages.optuna.dist
  • python311Packages.orange3
  • python311Packages.orange3.dist
  • python311Packages.plotly
  • python311Packages.plotly.dist
  • python311Packages.pyannote-audio
  • python311Packages.pyannote-audio.dist
  • python311Packages.pyannote-pipeline
  • python311Packages.pyannote-pipeline.dist
  • python311Packages.reflex
  • python311Packages.reflex.dist
  • python311Packages.scikit-tda
  • python311Packages.scikit-tda.dist
  • python311Packages.skytemple-dtef
  • python311Packages.skytemple-dtef.dist
  • python311Packages.skytemple-files
  • python311Packages.skytemple-files.dist
  • python311Packages.skytemple-ssb-debugger
  • python311Packages.skytemple-ssb-debugger.dist
  • python311Packages.synergy
  • python311Packages.synergy.dist
  • python311Packages.textnets
  • python311Packages.textnets.dist
  • python312Packages.ax-platform
  • python312Packages.ax-platform.dist
  • python312Packages.catboost
  • python312Packages.catboost.dist
  • python312Packages.chart-studio
  • python312Packages.chart-studio.dist
  • python312Packages.compressai
  • python312Packages.compressai.dist
  • python312Packages.cufflinks
  • python312Packages.cufflinks.dist
  • python312Packages.dash
  • python312Packages.dash-bootstrap-components
  • python312Packages.dash-bootstrap-components.dist
  • python312Packages.dash.dist
  • python312Packages.experiment-utilities
  • python312Packages.experiment-utilities.dist
  • python312Packages.explorerscript
  • python312Packages.explorerscript.dist
  • python312Packages.igraph
  • python312Packages.igraph.dist
  • python312Packages.kaleido
  • python312Packages.kaleido.dist
  • python312Packages.kmapper
  • python312Packages.kmapper.dist
  • python312Packages.leidenalg
  • python312Packages.leidenalg.dist
  • python312Packages.meshtastic
  • python312Packages.meshtastic.dist
  • python312Packages.optuna
  • python312Packages.optuna.dist
  • python312Packages.orange3
  • python312Packages.orange3.dist
  • python312Packages.plotly
  • python312Packages.plotly.dist
  • python312Packages.pyannote-audio
  • python312Packages.pyannote-audio.dist
  • python312Packages.pyannote-pipeline
  • python312Packages.pyannote-pipeline.dist
  • python312Packages.reflex
  • python312Packages.reflex.dist
  • python312Packages.scikit-tda
  • python312Packages.scikit-tda.dist
  • python312Packages.skytemple-dtef
  • python312Packages.skytemple-dtef.dist
  • python312Packages.skytemple-files
  • python312Packages.skytemple-files.dist
  • python312Packages.skytemple-ssb-debugger
  • python312Packages.skytemple-ssb-debugger.dist
  • python312Packages.synergy
  • python312Packages.synergy.dist
  • quark-engine
  • quark-engine.dist
  • skytemple
  • skytemple.dist
  • whisper-ctranslate2
  • whisper-ctranslate2.dist

aiomultiprocess failed to build, causing theharvester to fail. It's also failing on master. See my fix here: #339823

@natsukium natsukium merged commit 41120cb into NixOS:master Sep 5, 2024
28 of 30 checks passed
@Pandapip1 Pandapip1 deleted the init-python3packages-kaleido branch September 5, 2024 16:18
@Pandapip1 Pandapip1 added the backport release-24.05 Backport PR automatically label Sep 5, 2024
Copy link
Contributor

github-actions bot commented Sep 5, 2024

Backport failed for release-24.05, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin release-24.05
git worktree add -d .worktree/backport-339136-to-release-24.05 origin/release-24.05
cd .worktree/backport-339136-to-release-24.05
git switch --create backport-339136-to-release-24.05
git cherry-pick -x 25ad2d11ec45cbc6c908b78de63ad49aba401f9f 534c90d223c32bf88b7aba3399fa69936c25fbf6 46c1ce890a749f6c17b07ce87237e7ebccf5af42 6c9867637668f00d0e7f8dac0f3312c525500c9d 7edc3b600d3e35e422bed873b9956797aeec801b

@Pandapip1
Copy link
Contributor Author

Manually backported: #339839

@sarahec
Copy link
Contributor

sarahec commented Sep 5, 2024

Here's a weirdness: it builds fine on Darwin when run as a stand-alone build. The error you saw (essentially "we don't know how to find the system UI configuration") may be due to the building machine being headless.

Recommendation: disable the failing test. It passes when run on a desktop/laptop.

@emilazy
Copy link
Member

emilazy commented Sep 5, 2024

It’s just the Nix sandbox forbidding access to OS stuff. Happens a lot.

@Pandapip1
Copy link
Contributor Author

Unfortunately I don't have access to a darwin machine to debug this. I suspect it might be libGL-related?

@sarahec
Copy link
Contributor

sarahec commented Sep 5, 2024

Unfortunately I don't have access to a darwin machine to debug this. I suspect it might be libGL-related?

I think @emilazy has identified the issue. I'm attempting to verify it on a Darwin machine so we can disable the appropriate tests.

@Pandapip1
Copy link
Contributor Author

I think @emilazy has identified the issue.

Yea, I was just noting that libGL is one of the few impurities left in nixpkgs.

@emilazy
Copy link
Member

emilazy commented Sep 5, 2024

I’m not sure what you mean; libGL isn’t impure. On Darwin, it’s the same as darwin.apple_sdk.frameworks.OpenGL.

@eclairevoyant
Copy link
Contributor

Loading libGL is done impurely in NixOS, because nvidia.
(Not sure if the error is related since I can't find it atm.)

@emilazy
Copy link
Member

emilazy commented Sep 5, 2024

I wouldn’t quite call that an impurity. libGL itself is libglvnd, a perfectly pure, standard, Nix‐packaged library. That library happens to delegate to a system OpenGL driver at runtime, which I would say makes it late‐bound, rather than impure per se; the properties of the Nix build aren’t being violated at any point.

Anyway, that’s not too relevant to macOS, where every system framework is built inside Nix against purely‐packaged stubs listing the available symbols, and which is then linked against the OS‐level framework code at runtime by the dynamic loader. (So in a sense, every system library is like libGL on Darwin.)

@Pandapip1
Copy link
Contributor Author

I wouldn’t quite call that an impurity. libGL itself is libglvnd, a perfectly pure, standard, Nix‐packaged library. That library happens to delegate to a system OpenGL driver at runtime, which I would say makes it late‐bound, rather than impure per se; the properties of the Nix build aren’t being violated at any point.

libglvnd's .sos have RPATHs that contain paths outside of /nix/store.

@sarahec
Copy link
Contributor

sarahec commented Sep 5, 2024

I think @emilazy has identified the issue. I'm attempting to verify it on a Darwin machine so we can disable the appropriate tests.

I can't get the build to break for me on Darwin, even with sandboxing enabled. Should I file a PR that conditionally disables the passthrough test on Darwin and removes the broken flag?

@Pandapip1
Copy link
Contributor Author

I can't get the build to break for me on Darwin, even with sandboxing enabled. Should I file a PR that conditionally disables the passthrough test on Darwin and removes the broken flag?

Go right ahead!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants