-
-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug & Feature Request: Python bindings support pyproject.toml, conflicts with setuptools & setup.py #8069
Comments
@latot What do you mean by the above ? How to configure that ? |
Well... is weird, actually setuptools blocks imports, that means you can't import anything, or, well you can run the import statement but will always says there is no module, except if you pass the setup statement with the The weird part, is "block the imports" is right now the default behavior...., you should not do anything to activate that... I have the next ideas why this:
|
This is an example of #This will fails with no numpy module, in a try will trigger the catch
import numpy
setup(
...
setup_requires(["numpy"]),
...
)
#This one will works fine
import numpy Here I asked on setuptools, where says the solution for this is using Thx! |
@latot yes, If there is interest I could try to provide a patch. I would also suggest to move as much as possible form
AFAIK |
Hi, I used the wrong word sorry, is not deprecated, now is legacy, you can check that on the docs of https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/ I changed the word on the main post, from deprecated to legacy to be clear. I would like to do a patch, but right now I'm out of time, debug this issue was really slow D: |
That would be welcome. But cf my question at pypa/setuptools#3971 (reply in thread) Hum, wondering if having numpy as a required dependency wouldn't make things simpler. I presume most people who use GDAL Python bindings enable numpy. Perhaps something to raise on the mailing list to see if there are reactions to such proposal. |
I think that would be great too, have the new way and skip problems now and in the future.
I think that would do a lot of things simpler, at least form my perspective.
So I think would be great! |
I think that the use case of an optional build dependency is not addressed by any of the build backends AFAIK. By the way one could face the problem form a different perspective.
or
In this case we would have a (mandatory) build dependency on numpy and just an optional dependency at runtime. The problem at this point is that currently binary wheels for GDAL are not produced so the above commands will always result the download of numpy. If the isolated build is used then numpy will not go in the user environment unless explicitly requested ... but I need to check this.
for this, I think you need to use
I don't think I've ever heard about people using GDAL Python bindings without numpy. |
@avalentino Great to know all that. An extra part, that is described above, is that setuptools, right now, only allows you to import a module when is declared in setup and import after the call: #8069 (comment) , is more well explained in the main post. So even if you have numpy installed, you will not be allowed to import numpy with the actual With the actual behavior of setuptools even with |
|
@rouault Can you try upgrading pip + setuptools? remember no wheels O.o |
Hi!, I was able to reproduce it in debian!
|
ok I can now reproduce "pip install numpy" being ignored by "pip install gdal", when not installing |
@rouault I just stepped through this problem and would like to echo @avalentino's summary (#8069 (comment)) – especially the bit about Ultimately I agree that this is a flaw in the Python packaging ecosystem. Ideally the Python Packaging Authority would have provided one of these paths:
However, I am also struggling to see why a user would want to install the The current method of installing
In order to install $ pip install package --only-deps to install only So, if I invoke these commands to install
|
Mailing list probed about requiring numpy: https://lists.osgeo.org/pipermail/gdal-dev/2023-December/058064.html |
Great, thank you! |
Some related background: https://peps.python.org/pep-0517/ |
…ult, unless the GDAL_PYTHON_BINDINGS_WITHOUT_NUMPY env var is set (refs OSGeo#8069)
cf #8926 for hopefully a resolution of this issue |
…nt (#8926) Fixes #8069. Hopefully a compromise that will satisfy most needs... - add a pyproject.toml with numpy as a build requirement - make absence of numpy at build time an error by default, unless the GDAL_PYTHON_BINDINGS_WITHOUT_NUMPY env var is set. - despite numpy being a build requirement, numpy remains an optional dependency at install time if users just install with "pip install gdal" (for folks that would do vector only operations with GDAL) - for people needing numpy support, they have to use "pip install gdal[numpy]". * swig/python/CMakeLists.txt: check return value of execute_process() at various places
I dont want to open this feature but after going back to gdal 3.6.2 with python3.11 and this answer solved the issue for me. I tried every suggestion here and also everthing i could find on the internet. So maybe this will help someone in the future. https://gis.stackexchange.com/a/465888 My adapation for my Dockerfile
|
Hi all, this issue started with #8024
Bug
Oks, lets start, first
setup.py
I has two steps.
This workflow has worked in the time, but the reason now why is failing is the next one, actually, setuptools does not allow any more import any module that is not declared in
setup_required
param of the setup statement, and thesetup
statement must be executed in order to be available to import that module, and will only by available after the setup statement.This causes that when the python module needs to be compiled the numpy import will fails, and it will trigger the except statement saying to GDAL that numpy is not installed, because the numpy check needs to be executed before the setup statement.
So, the rules that are in conflict right now:
Numpy must be checked before the setup statement
Numpy can only be imported after setup statement with the right params to allow the import
Is hard to see this problem?
Well... yes, lets go step by step.
In debian or ubuntu there is the package libgdal-python, which install all precompiled, so there is no need to do nothing of the above mentioned. Even is probable to python-numpy is installed as a dep of gdal or any other package, so any try to use the system python gdal will just works.
With venv is different, there will be no gdal bindings, no numpy from the start, and actually the warning of no numpy is just that, usually passes with all the text without someone notice it happens.
So, in order to find this bug, you need a setuptools with the actual rule of blocked imports, use venv to isolate a system, and then run with
pip install -vvv
without wheels to just read the warning that tells you numpy was not detected.... and if you know numpy is installed, just now you know "here is a problem"Solution
Oks, I have asked on python matrix, pip dev, setuptools comunity.... is there a way to solve this? yes.
But first, this is not directly a problem on GDAL, actually setuptools has a lot of problems, one of them is that you are not able to handle the setup env.
Here is where the solution comes, use
pyproject.toml
, this replacessetup.py
and have the features to be able to handle this cases, evensetuptools
now is legacy due to the problems it has.They send me this link: https://godatadriven.com/blog/a-practical-guide-to-setuptools-and-pyproject-toml/
There is this one too: https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/
I was not able to find a solution with setuptools, but maybe this way is safer for mid and long term.
Special thx @rouault with advice and help debugging this!
Thx!
The text was updated successfully, but these errors were encountered: