-
Notifications
You must be signed in to change notification settings - Fork 172
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pixi as a main development driver and pixi env -e
behavior
#2266
Comments
Hi @adrinjalali, I'm not sure what you are requesting, do you mean Also I'm not aware of a I believe you want to be able to run |
I'll try to describe some potential use cases that we encountered when developing that require a bit more flexibility than the CI case. One use case is related to being able to install a library without necessarily impacting the lock file. Installing a dependency tracked in the lock fileWhen getting a bug fix report, one might need to switch the version of a dependency. The version is therefore not necessarily linked to a specific environment. One potential flow would be: pixi shell -e dev
conda install scikit-learn~=1.4.0 # we might install via `pip` as well
pytest my_package While deactivating/re-activating the environment, I would expect either (or both) of the following behaviors:
Current state: Currently, it looks like you can have pixi shell -e dev
conda install scipy~=1.13.1 We will also get a downgrade of:
Installing a new dependency without tracking it in the lock fileSometimes, one would like to install a dependency that should not be tracked with the lock file, e.g., Current state: Similarly to above, you can intend to install an additional package if pixi shell -e dev
conda install ipython The interesting part here is that while installed with Installing several packages from sourceSometimes, we end up installing several libraries from source and switching branches locally. While From my thoughts, it could look like something as proposed here: https://github.com/rgommers/pixi-dev-scipystack but where I would expect to have a top-level ConclusionsI completely agree that any of the use cases above is just stretching the scope of what |
Thank you for the great write up @glemaitre. This is really good food for thought! I'll definitely pass it to the team! We've been pretty conservative with features that allow you to create "broken" environments but "Pixi is build for development" so we need to find a way to serve as much users as possible. I would like to get some users stories from your input. Would these be correct:
If these stories fit the need, we can start brainstorming implementation ideas. |
Those user stories sound good to me. What I would add is:
|
@adrinjalali, just checking, but doesn't your command prompt tell you the environment you are in? |
I have to double check my zsh config but I don' get the right info: @adrinjalali might actually be a bug :). Edit: So it seems that there is some side-effect when
I would amend slightly this user story mentioning that I have the option to "leave the environment or to completely reset it". Otherwise, both user stores look the good. |
This comment has been minimized.
This comment has been minimized.
I've run into these workflow issues as well, but have come to different conclusions. To start with, an observation that if you're trying to use a conda-like workflow with imperative temporary modifications to an environment, you're really trying to fight one of the most fundamental design decisions in Pixi - from https://pixi.sh/latest/features/environment/#structure: Pixi will always make sure the environment is in sync with the I'd say that the real user story for the first two items is actually not "let me do the conda-like thing" but "How can I change a version of a dependency or temporarily add a new dependency easily, in order to effectively debug an issue reported against my library for that specific version of the dependency?" I've so far found it not too onerous to just modify Taking a step back, there are two types of debugging exercises here I think:
The first two examples in @glemaitre's first post are of type (1). I'd say that Pixi currently supports this fine; working declaratively isn't really harder than working imperatively in a throw-away conda env with The third example (install multiple package from source) is harder, as is my "apply a patch" example. For the former, perhaps the workspace concept can help once that lands. For the inherently type (2) cases though, I've concluded that that is where I still need to keep one of |
Thanks @rgommers for the feedback. I second the reformulation :)
I think that I slightly disagree with a "it depends". If I'm the target user then I completely agree because I know how to use Apart from this point, I'm really aligned with @rgommers thought and thinking that the "workspace" concept might seem the most natural way to deal with multiple from-source install. |
Yeah fair enough, newer devs will struggle. I'd say both are probably fairly hard for newcomers though. The imperative way is to do
That's hard indeed. I'd document it only via |
Something that makes me quite uneasy about this is that doing a There's this coupling of files on the repo and changes to the local workspace env which I'd like not to have. |
This is kind of the main idea of the pixi design, unless you know exactly what you're doing it's often easier to |
People working on the same project, have vastly different environments, and that's a good thing. For instance, I might want to have Or I might not want to bother with a lot of optional dependencies and have those tests skipped, while others might be working on them. Having different environments on different contributor machines is a good thing, since it uses the diversity of the contributor base to test the code base against a larger pool of potential environments out there used by our end users. This means the environments shouldn't be strictly dictated by the repo itself, and contributors should have flexibility on what they do with their envs. This means we'd need to not have a pixi config on the repo and have it git ignored. Or have a way to ignore the repo's pixi config and each user to have a pixi config per project in their home folder somewhere and tell pixi to use those instead. This doesn't sound ideal to me. |
It's not unambiguously a good thing, it has important pros and cons. A pro is what you say: more testing in varied envs. Cons are: it's much harder to set up such varied envs or write good contributor docs for it, for newer contributors there's more breakage when all they want to do is work on a pure Python PR, etc. I think Pixi fixing the cons above is a big gain. There will still be users wanting to use venvs, or conda/mamba, or spack, or docker, or whatever, so the pro you describe won't disappear completely just by adopting Pixi. My recommendation would be to set things up so that using Pixi is a good default for newcomers to your project, but optional. Turning Pixi into working just like Conda is kinda what you're asking for, but can't really be the right answer. I actually quite like the |
I guess a difference between numpy and scikit-learn for instance, is that in numpy you don't really "install" the package in an environment, and you always have to do something (setting some paths) to use it, which When I work on numpy, I always have to run That is not to say that I think |
Problem description
Right now
pixi
as is, is great for reproducing work and environments and to be used in the CI. However, it's not suitable for everyday development and a part of that is a lack of ability to manipulate and persist an environment.pixi env -e
as is, activates an environment whose path includes a hash, which is quite confusing and not memorable, therefore not useful as remembering which env one is using. Also, the behavior of installing a package in that env, and then deactivating and reactivating it is somewhat unclear. There's--freeze
which also doesn't make it clear what happens the next time if the user forgets to have that flag when activating and env.It would be nice to have a way for using
pixi
that enables the developer to work with environments, somewhat similar to how one uses an environment w/opixi
.cc @glemaitre
The text was updated successfully, but these errors were encountered: