Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segfault when operating over strided inputs in free-threaded Python #2271

Open
andfoy opened this issue Jan 6, 2025 · 3 comments · May be fixed by #2273
Open

Segfault when operating over strided inputs in free-threaded Python #2271

andfoy opened this issue Jan 6, 2025 · 3 comments · May be fixed by #2273

Comments

@andfoy
Copy link

andfoy commented Jan 6, 2025

As part of the free-threaded compliance testing performed over at SciPy, we have discovered the following Pythran issue when covering scipy.stats scipy/scipy#22125:

PYTHON_GIL=0 pytest -x -v --parallel-threads=10 --iterations=40 scipy/stats/tests/test_hypotests.py::TestSomersD
============================================================= test session starts ==============================================================
platform linux -- Python 3.13.0, pytest-8.3.3, pluggy-1.5.0 -- /home/andfoy/anaconda3/envs/nogil/bin/python3.13
cachedir: .pytest_cache
hypothesis profile 'deterministic' -> database=None, deadline=None, print_blob=True, derandomize=True, suppress_health_check=[HealthCheck.data_too_large, HealthCheck.filter_too_much, HealthCheck.too_slow, HealthCheck.large_base_example, HealthCheck.function_scoped_fixture, HealthCheck.differing_executors]
rootdir: /home/andfoy/Documentos/Quansight/nogil/scipy
configfile: pytest.ini
plugins: hypothesis-6.118.8, run-parallel-0.2.1.dev0, order-1.3.0
collected 15 items                                                                                                                             

scipy/stats/tests/test_hypotests.py::TestSomersD::test_all_dtypes PASSED                                                                 [  6%]
scipy/stats/tests/test_hypotests.py::TestSomersD::test_views PASSED                                                                      [ 13%]
scipy/stats/tests/test_hypotests.py::TestSomersD::test_strided Fatal Python error: Aborted

Thread 0x0000704593e00640 (most recent call first):
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 359 in wait
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 394 in wait_for
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 764 in _wait
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 729 in wait
  File "/home/andfoy/Documentos/Quansight/nogil/pytest-run-parallel/src/pytest_run_parallel/plugin.py", line 57 in closure
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 992 in run
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1041 in _bootstrap_inner
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1012 in _bootstrap

Thread 0x000070459cc00640 (most recent call first):
  File "/home/andfoy/Documentos/Quansight/nogil/scipy/scipy/stats/_crosstab.py", line 204 in crosstab
  File "/home/andfoy/Documentos/Quansight/nogil/scipy/scipy/stats/_hypotests.py", line 886 in somersd
  File "/home/andfoy/Documentos/Quansight/nogil/scipy/scipy/stats/tests/test_hypotests.py", line 683 in pythranfunc
  File "/home/andfoy/Documentos/Quansight/nogil/scipy/scipy/_lib/_testutils.py", line 207 in test_strided
  File "/home/andfoy/Documentos/Quansight/nogil/pytest-run-parallel/src/pytest_run_parallel/plugin.py", line 59 in closure
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 992 in run
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1041 in _bootstrap_inner
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1012 in _bootstrap

Thread 0x000070459d600640 (most recent call first):
  File "/home/andfoy/Documentos/Quansight/nogil/scipy/scipy/stats/_hypotests.py", line 694 in _somers_d
  File "/home/andfoy/Documentos/Quansight/nogil/scipy/scipy/stats/_hypotests.py", line 886 in somersd
  File "/home/andfoy/Documentos/Quansight/nogil/scipy/scipy/stats/tests/test_hypotests.py", line 683 in pythranfunc
  File "/home/andfoy/Documentos/Quansight/nogil/scipy/scipy/_lib/_testutils.py", line 207 in test_strided
  File "/home/andfoy/Documentos/Quansight/nogil/pytest-run-parallel/src/pytest_run_parallel/plugin.py", line 59 in closure
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 992 in run
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1041 in _bootstrap_inner
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1012 in _bootstrap

Thread 0x000070459e000640 (most recent call first):
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/numpy/_core/_methods.py", line 73 in _all
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/numpy/testing/_private/utils.py", line 737 in func_assert_same_pos
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/numpy/testing/_private/utils.py", line 781 in assert_array_compare
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/numpy/testing/_private/utils.py", line 1676 in assert_allclose
  File "/home/andfoy/Documentos/Quansight/nogil/scipy/scipy/stats/tests/test_hypotests.py", line 684 in pythranfunc
  File "/home/andfoy/Documentos/Quansight/nogil/scipy/scipy/_lib/_testutils.py", line 207 in test_strided
  File "/home/andfoy/Documentos/Quansight/nogil/pytest-run-parallel/src/pytest_run_parallel/plugin.py", line 59 in closure
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 992 in run
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1041 in _bootstrap_inner
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1012 in _bootstrap

Thread 0x00007045a5600640 (most recent call first):
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 359 in wait
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 394 in wait_for
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 764 in _wait
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 729 in wait
  File "/home/andfoy/Documentos/Quansight/nogil/pytest-run-parallel/src/pytest_run_parallel/plugin.py", line 57 in closure
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 992 in run
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1041 in _bootstrap_inner
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1012 in _bootstrap

Thread 0x00007045a6000640 (most recent call first):
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 359 in wait
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 394 in wait_for
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 764 in _wait
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 729 in wait
  File "/home/andfoy/Documentos/Quansight/nogil/pytest-run-parallel/src/pytest_run_parallel/plugin.py", line 57 in closure
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 992 in run
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1041 in _bootstrap_inner
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1012 in _bootstrap

Current thread 0x00007045a4c00640 (most recent call first):
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/numpy/_core/fromnumeric.py", line 57 in _wrapfunc
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/numpy/_core/fromnumeric.py", line 506 in repeat
  File "/home/andfoy/Documentos/Quansight/nogil/scipy/scipy/_lib/_testutils.py", line 205 in test_strided
  File "/home/andfoy/Documentos/Quansight/nogil/pytest-run-parallel/src/pytest_run_parallel/plugin.py", line 59 in closure
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 992 in run
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1041 in _bootstrap_inner
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1012 in _bootstrap

Thread 0x000070459fe00640 (most recent call first):
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 359 in wait
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 394 in wait_for
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 764 in _wait
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 729 in wait
  File "/home/andfoy/Documentos/Quansight/nogil/pytest-run-parallel/src/pytest_run_parallel/plugin.py", line 57 in closure
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 992 in run
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1041 in _bootstrap_inner
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1012 in _bootstrap

Thread 0x000070459f400640 (most recent call first):
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 359 in wait
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 394 in wait_for
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 764 in _wait
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 729 in wait
  File "/home/andfoy/Documentos/Quansight/nogil/pytest-run-parallel/src/pytest_run_parallel/plugin.py", line 57 in closure
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 992 in run
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1041 in _bootstrap_inner
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1012 in _bootstrap

Thread 0x000070459ea00640 (most recent call first):
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 359 in wait
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 394 in wait_for
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 764 in _wait
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 729 in wait
  File "/home/andfoy/Documentos/Quansight/nogil/pytest-run-parallel/src/pytest_run_parallel/plugin.py", line 57 in closure
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 992 in run
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1041 in _bootstrap_inner
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1012 in _bootstrap

Thread 0x0000704658f9f440 (most recent call first):
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/threading.py", line 1092 in join
  File "/home/andfoy/Documentos/Quansight/nogil/pytest-run-parallel/src/pytest_run_parallel/plugin.py", line 82 in inner
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/_pytest/python.py", line 159 in pytest_pyfunc_call
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/_pytest/python.py", line 1627 in runtest
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/_pytest/runner.py", line 174 in pytest_runtest_call
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/_pytest/runner.py", line 242 in <lambda>
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/_pytest/runner.py", line 341 in from_call
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/_pytest/runner.py", line 241 in call_and_report
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/_pytest/runner.py", line 132 in runtestprotocol
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/_pytest/runner.py", line 113 in pytest_runtest_protocol
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/_pytest/main.py", line 362 in pytest_runtestloop
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/_pytest/main.py", line 337 in _main
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/_pytest/main.py", line 283 in wrap_session
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/_pytest/main.py", line 330 in pytest_cmdline_main
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/_pytest/config/__init__.py", line 175 in main
  File "/home/andfoy/anaconda3/envs/nogil/lib/python3.13t/site-packages/_pytest/config/__init__.py", line 201 in console_main
  File "/home/andfoy/anaconda3/envs/nogil/bin/pytest", line 8 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, cython.cimports.libc.math, scipy._lib._ccallback_c, scipy._lib._fpumode, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._cython_nnls, scipy._lib._uarray._uarray, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.interpolate._dierckx, scipy.interpolate._ppoly, scipy.interpolate._interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.interpolate._bspl, scipy.special.cython_special, scipy.stats._stats, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._biasedurn, scipy.stats._stats_pythran, scipy.stats._levy_stable.levyst, scipy.stats._ansari_swilk_statistics, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.ndimage._nd_image, scipy.ndimage._rank_filter_1d, _ni_label, scipy.ndimage._ni_label (total: 96)
Abortado (`core' generado)

Apparently, the segfault is related to memory pointer deallocation (via free), which is highlighted after running the following reproducing script:

Reproducing script 
import functools
import numpy as np
import threading
from scipy import stats

x0 = np.arange(10)
x1 = np.arange(10)
partialfunc = functools.partial(stats.somersd, alternative='two-sided')

n_workers = 10
n_iterations = 100
barrier = threading.Barrier(n_workers)

def closure(*args, **kwargs):
    barrier.wait()
    for _ in range(n_iterations):
        x = np.repeat(x0, 2, axis=0)[::2]
        y = np.repeat(x1, 2, axis=0)[::2]
        partialfunc(x, y)

workers = []
for _ in range(0, n_workers):
    worker_kwargs = {}
    workers.append(
        threading.Thread(target=closure, args=tuple(), kwargs=worker_kwargs)
    )

for worker in workers:
    worker.start()

for worker in workers:
    worker.join()
PYTHON_GIL=0 python ../pythran_reproducer.py 
free(): invalid size
Abortado (`core' generado)

PYTHON_GIL=0 python ../pythran_reproducer.py 
python: malloc.c:2617: sysmalloc: Assertion `(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.
Exception in thread Thread-5 (closure):
Exception in thread Thread-1 (closure):
Abortado (`core' generado)

The offending pythran functions (_Aij, _Dij and _a_ij_Aij_Dij2) are defined on the following file: https://github.com/scipy/scipy/blob/main/scipy/stats/_stats_pythran.py, whilst somersd is defined over at https://github.com/scipy/scipy/blob/a32bc49b738db04e5823b676be40bd8dab617f8c/scipy/stats/_hypotests.py#L678
@serge-sans-paille
Copy link
Owner

I built a local version of python 13 with --disable-gil and I can reproduce, let's dig...

serge-sans-paille added a commit that referenced this issue Jan 11, 2025
@serge-sans-paille
Move GIl manipulation right after module creation
9dc7a79 
So that it also affects objects created at module loading time.

This seems to fix #2271
@serge-sans-paille serge-sans-paille linked a pull request Jan 11, 2025 that will close this issue
Open
serge-sans-paille added a commit that referenced this issue Jan 11, 2025
@serge-sans-paille
Move GIL manipulation right after module creation
9d88807 
So that it also affects objects created at module loading time.

This seems to fix #2271
@serge-sans-paille
Copy link
Owner

Could you confirm #2273 fixes the issue? It does on my setup.

@serge-sans-paille
Copy link
Owner

@andfoy : gentle ping :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Move GIL manipulation right after module creation serge-sans-paille/pythran
2 participants
@serge-sans-paille @andfoy