Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_dbm and test_dbm_ndbm segfaults in M1 Mac with Homebrew gdbm 1.23 #89452

Closed
pablogsal opened this issue Sep 25, 2021 · 12 comments
Closed

test_dbm and test_dbm_ndbm segfaults in M1 Mac with Homebrew gdbm 1.23 #89452

pablogsal opened this issue Sep 25, 2021 · 12 comments
Labels
3.9 only security fixes 3.10 only security fixes 3.11 only security fixes OS-mac type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@pablogsal
Copy link
Member

pablogsal commented Sep 25, 2021

BPO 45289
Nosy @ronaldoussoren, @ned-deily, @corona10, @pablogsal

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-09-25.19:56:32.069>
labels = ['OS-mac', '3.10', '3.9', 'type-crash', '3.11']
title = 'test_dbm and test_dbm_ndbm segfaults in M1 Mac'
updated_at = <Date 2021-09-26.05:38:13.631>
user = 'https://github.com/pablogsal'

bugs.python.org fields:

activity = <Date 2021-09-26.05:38:13.631>
actor = 'corona10'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['macOS']
creation = <Date 2021-09-25.19:56:32.069>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 45289
keywords = []
message_count = 2.0
messages = ['402643', '402645']
nosy_count = 4.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'corona10', 'pablogsal']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue45289'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Hand-linked PRs

Linked PRs

@pablogsal
Copy link
Member Author

pablogsal commented Sep 25, 2021

0:04:14 load avg: 3.86 [141/427/1] test_dbm crashed (Exit code -11)
Fatal Python error: Segmentation fault
Current thread 0x0000000102e2bd40 (most recent call first):
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_dbm.py", line 58 in keys_helper
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_dbm.py", line 143 in read_helper
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_dbm.py", line 74 in test_anydbm_creation
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/case.py", line 547 in \_callTestMethod
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/case.py", line 591 in run
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/case.py", line 646 in \_\_call__
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/suite.py", line 122 in run
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/suite.py", line 84 in \_\_call__
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/suite.py", line 122 in run
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/suite.py", line 84 in \_\_call__
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/suite.py", line 122 in run
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/suite.py", line 84 in \_\_call__
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/runner.py", line 197 in run
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/support/__init__.py", line 992 in \_run_suite
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/support/__init__.py", line 1118 in run_unittest
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/libregrtest/runtest.py", line 261 in \_test_module
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/libregrtest/runtest.py", line 297 in \_runtest_inner2
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/libregrtest/runtest.py", line 340 in \_runtest_inner
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/libregrtest/runtest.py", line 202 in \_runtest
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/libregrtest/runtest.py", line 245 in runtest
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/libregrtest/runtest_mp.py", line 83 in run_tests_worker
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/libregrtest/main.py", line 678 in \_main
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/libregrtest/main.py", line 658 in main
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/libregrtest/main.py", line 736 in main
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/regrtest.py", line 43 in \_main
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/regrtest.py", line 47 in \<module\>
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/runpy.py", line 86 in \_run_code
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/runpy.py", line 196 in \_run_module_as_main
Extension modules: \_testcapi (total: 1)
0:04:15 load avg: 3.87 [142/427/1] test_c_

The backtrace according to lldb:

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x10365c000)
  * frame #0: 0x0000000182bc6994 libsystem_platform.dylib`_platform_memmove + 84
    frame #1: 0x00000001000b71d0 python.exe`PyBytes_FromStringAndSize(str="c", size=4294967297) at bytesobject.c:165:5
    frame #2: 0x000000010360b234 _dbm.cpython-311d-darwin.so`_dbm_dbm_keys_impl(self=0x00000001034b3ca0, cls=0x000000010088a580) at _dbmmodule.c:249:16
    frame #3: 0x000000010360af34 _dbm.cpython-311d-darwin.so`_dbm_dbm_keys(self=0x00000001034b3ca0, cls=0x000000010088a580, args=0x00000001007991d8, nargs=0, kwnames=0x0000000000000000) at _dbmmodule.c.h:46:20
    frame #4: 0x00000001000dca8c python.exe`method_vectorcall_FASTCALL_KEYWORDS_METHOD(func=0x00000001034db290, args=0x00000001007991d0, nargsf=9223372036854775809, kwnames=0x0000000000000000) at descrobject.c:367:24
    frame #5: 0x0000000100252374 python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100808b30, callable=0x00000001034db290, args=0x00000001007991d0, nargsf=9223372036854775809, kwnames=0x0000000000000000) at abstract.h:114:11
    frame #6: 0x000000010024fc10 python.exe`PyObject_Vectorcall(callable=0x00000001034db290, args=0x00000001007991d0, nargsf=9223372036854775809, kwnames=0x0000000000000000) at abstract.h:123:12
    frame #7: 0x000000010024fd70 python.exe`call_function(tstate=0x0000000100808b30, pp_stack=0x000000016fdc8aa0, oparg=1, kwnames=0x0000000000000000, use_tracing=0) at ceval.c:6418:13
    frame #8: 0x000000010024a518 python.exe`_PyEval_EvalFrameDefault(tstate=0x0000000100808b30, frame=0x0000000100799150, throwflag=0) at ceval.c:4568:19
    frame #9: 0x000000010023a0f0 python.exe`_PyEval_EvalFrame(tstate=0x0000000100808b30, frame=0x0000000100799150, throwflag=0) at pycore_ceval.h:46:12
    frame #10: 0x000000010023a00c python.exe`_PyEval_Vector(tstate=0x0000000100808b30, con=0x00000001034e0320, locals=0x0000000000000000, args=0x0000000100799118, argcount=2, kwnames=0x0000000000000000) at ceval.c:5609:24
    frame #11: 0x00000001000ccbe4 python.exe`_PyFunction_Vectorcall(func=0x00000001034e0310, stack=0x0000000100799118, nargsf=9223372036854775810, kwnames=0x0000000000000000) at call.c:342:16
    frame #12: 0x0000000100252374 python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100808b30, callable=0x00000001034e0310, args=0x0000000100799118, nargsf=9223372036854775810, kwnames=0x0000000000000000) at abstract.h:114:11
    frame #13: 0x000000010024fc10 python.exe`PyObject_Vectorcall(callable=0x00000001034e0310, args=0x0000000100799118, nargsf=9223372036854775810, kwnames=0x0000000000000000) at abstract.h:123:12
    frame #14: 0x000000010024fd70 python.exe`call_function(tstate=0x0000000100808b30, pp_stack=0x000000016fdca950, oparg=2, kwnames=0x0000000000000000, use_tracing=0) at ceval.c:6418:13
    frame #15: 0x000000010024a518 python.exe`_PyEval_EvalFrameDefault(tstate=0x0000000100808b30, frame=0x00000001007990a8, throwflag=0) at ceval.c:4568:19
    frame #16: 0x000000010023a0f0 python.exe`_PyEval_EvalFrame(tstate=0x0000000100808b30, frame=0x00000001007990a8, throwflag=0) at pycore_ceval.h:46:12
    frame #17: 0x000000010023a00c python.exe`_PyEval_Vector(tstate=0x0000000100808b30, con=0x00000001034e0c10, locals=0x0000000000000000, args=0x0000000100799080, argcount=2, kwnames=0x0000000000000000) at ceval.c:5609:24
    frame #18: 0x00000001000ccbe4 python.exe`_PyFunction_Vectorcall(func=0x00000001034e0c00, stack=0x0000000100799080, nargsf=9223372036854775810, kwnames=0x0000000000000000) at call.c:342:16
    frame #19: 0x0000000100252374 python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100808b30, callable=0x00000001034e0c00, args=0x0000000100799080, nargsf=9223372036854775810, kwnames=0x0000000000000000) at abstract.h:114:11
    frame #20: 0x000000010024fc10 python.exe`PyObject_Vectorcall(callable=0x00000001034e0c00, args=0x0000000100799080, nargsf=9223372036854775810, kwnames=0x0000000000000000) at abstract.h:123:12
    frame #21: 0x000000010024fd70 python.exe`call_function(tstate=0x0000000100808b30, pp_stack=0x000000016fdcc800, oparg=2, kwnames=0x0000000000000000, use_tracing=0) at ceval.c:6418:13
    frame #22: 0x000000010024a518 python.exe`_PyEval_EvalFrameDefault(tstate=0x0000000100808b30, frame=0x0000000100799018, throwflag=0) at ceval.c:4568:19
    frame #23: 0x000000010023a0f0 python.exe`_PyEval_EvalFrame(tstate=0x0000000100808b30, frame=0x0000000100799018, throwflag=0) at pycore_ceval.h:46:12
    frame #24: 0x000000010023a00c python.exe`_PyEval_Vector(tstate=0x0000000100808b30, con=0x00000001034e0530, locals=0x0000000000000000, args=0x0000000100798ff0, argcount=1, kwnames=0x0000000000000000) at ceval.c:5609:24
......

The last 3 frames:

frame #1: 0x00000001000b71d0 python.exe`PyBytes_FromStringAndSize(str="c", size=4294967297) at bytesobject.c:165:5
   162 	    if (str == NULL)
   163 	        return (PyObject *) op;
   164
-> 165 	    memcpy(op->ob_sval, str, size);
   166 	    /* share short strings */
   167 	    if (size == 1) {
   168 	        struct _Py_bytes_state *state = get_bytes_state();
(lldb)
frame #2: 0x000000010360b234 _dbm.cpython-311d-darwin.so`_dbm_dbm_keys_impl(self=0x00000001034b3ca0, cls=0x000000010088a580) at _dbmmodule.c:249:16
   246 	    }
   247 	    for (key = dbm_firstkey(self->di_dbm); key.dptr;
   248 	         key = dbm_nextkey(self->di_dbm)) {
-> 249 	        item = PyBytes_FromStringAndSize(key.dptr, key.dsize);
   250 	        if (item == NULL) {
   251 	            Py_DECREF(v);
   252 	            return NULL;
(lldb)
frame #3: 0x000000010360af34 _dbm.cpython-311d-darwin.so`_dbm_dbm_keys(self=0x00000001034b3ca0, cls=0x000000010088a580, args=0x00000001007991d8, nargs=0, kwnames=0x0000000000000000) at _dbmmodule.c.h:46:20
   43  	        )) {
   44  	        goto exit;
   45  	    }
-> 46  	    return_value = _dbm_dbm_keys_impl(self, cls);
   47
   48  	exit:
   49  	    return return_value;

The segfault happens here:

frame #0: 0x0000000182bc6994 libsystem_platform.dylib`_platform_memmove + 84
libsystem_platform.dylib`_platform_memmove:
->  0x182bc6994 <+84>: ldnp   q0, q1, [x1]
    0x182bc6998 <+88>: add    x1, x1, #0x20             ; =0x20
    0x182bc699c <+92>: subs   x2, x2, #0x20             ; =0x20
    0x182bc69a0 <+96>: b.hi   0x182bc698c               ; <+76>

@pablogsal pablogsal added tests Tests in the Lib/test dir 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes type-crash A hard crash of the interpreter, possibly with a core dump OS-mac and removed tests Tests in the Lib/test dir labels Sep 25, 2021
@pablogsal
Copy link
Member Author

@pablogsal pablogsal changed the title test_gdbm segfaults in M1 Mac test_dbm and test_dbm_ndbm segfaults in M1 Mac Sep 25, 2021
@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
@ambv
Copy link
Contributor

ambv commented May 2, 2022

Looking at macOS native crash reporting, the issue is in this line of _dbm_dbm_keys_impl within the _dbmmodule.c:

    for (key = dbm_firstkey(self->di_dbm); key.dptr;
         key = dbm_nextkey(self->di_dbm)) {
        item = PyBytes_FromStringAndSize(key.dptr, key.dsize);  // <<<<<<<<<<
        if (item == NULL) {
            Py_DECREF(v);
            return NULL;
        }

Crash report:

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib            0x199490db8 __pthread_kill + 8
1   libsystem_pthread.dylib           0x1994c5ee0 pthread_kill + 288
2   libsystem_c.dylib                 0x1993c8674 raise + 32
3   Python                            0x10512fb00 faulthandler_fatal_error + 356 (faulthandler.c:375)
4   libsystem_platform.dylib          0x1994dd4c4 _sigtramp + 56
5   Python                            0x104ed5df0 PyBytes_FromStringAndSize + 272 (bytesobject.c:142)
6   _dbm.cpython-311d-darwin.so       0x104b9b494 _dbm_dbm_keys_impl + 220 (_dbmmodule.c:255)

@ambv
Copy link
Contributor

ambv commented May 2, 2022

A minimal reproduction is:

import dbm.ndbm
from pathlib import Path

db_path = Path("/tmp/test.dbm")
db_path.unlink(missing_ok=True)
f = dbm.ndbm.open(db_path, 'c')
print("Underlying lib:", dbm.ndbm.library)
print("f is an", repr(f))
print("The next line segfaults:")
f.keys()

This crashes with:

Underlying lib: GNU gdbm
f is an <_dbm.dbm object at 0x105a8fe80>
The next line segfaults:
Fatal Python error: Segmentation fault

Current thread 0x0000000105794580 (most recent call first):
  File "/Volumes/RAMDisk/cpython/_test.py", line 10 in <module>
fish: Job 1, './python.exe -X faulthandler _t…' terminated by signal SIGSEGV (Address boundary error)

@ambv
Copy link
Contributor

ambv commented May 2, 2022

Note that swapping dbm.ndbm for dbm.gnu causes the example to run without crashing:

import dbm.gnu
from pathlib import Path

db_path = Path("/tmp/test.dbm")
db_path.unlink(missing_ok=True)
f = dbm.gnu.open(db_path, 'c')
print("f is an", repr(f))
print("The next line runs fine:")
f.keys()

This just outputs:

❯ ./python.exe -X faulthandler _test.py                                                                                                                                              
f is an <_gdbm.gdbm object at 0x103afae90>
The next line runs fine:
❯

So looks like there is something off with the ndbm compatibility layer of gdbm on M1 Macs.

@ambv
Copy link
Contributor

ambv commented May 2, 2022

This is on Homebrew gdbm 1.23.

@ambv
Copy link
Contributor

ambv commented May 2, 2022

Apparently this runs fine on the MacPorts version of gdbm 1.23.

@ambv ambv changed the title test_dbm and test_dbm_ndbm segfaults in M1 Mac test_dbm and test_dbm_ndbm segfaults in M1 Mac with Homebrew gdbm 1.23 May 2, 2022
@ambv
Copy link
Contributor

ambv commented May 3, 2022

Ooohhh, I get it. The reason Homebrew fails and MacPorts works is that the former provides the ndbm-compat interface while the latter doesn't (by default).

Python discovers the ndbm-compat interface but then chooses the wrong header, causing segfaults. You see, the macOS SDK ships with a ndbm.h which isn't really usable (see GH-74573). Homebrew renames gdbm's ndbm.h to the Debian-compatible gdbm-ndbm.h which is picked up by the configure script:

checking ndbm.h usability... yes
checking ndbm.h presence... yes
checking for ndbm.h... yes
checking for dbm_open in -lndbm... no
checking for dbm_open in -lgdbm_compat... yes
checking gdbm/ndbm.h usability... no
checking gdbm/ndbm.h presence... no
checking for gdbm/ndbm.h... no
checking gdbm-ndbm.h usability... yes
checking gdbm-ndbm.h presence... yes
checking for gdbm-ndbm.h... yes
checking for dbm_open in -lgdbm_compat... (cached) yes
checking db.h usability... yes
checking db.h presence... yes
checking for db.h... yes
checking for libdb... yes
checking for --with-dbmliborder... ndbm:gdbm:bdb

As you see in the configure output above, the ndbm.h is not usable (-lndbm has no dbm_open) so the _dbmmodule.c is built through -lgdbm_compat working. But then due to --with-dbmliborder shown above, the invalid incompatible ndbm.h is included first in _dbmmodule.c:

#if defined(USE_NDBM)
#include <ndbm.h>
static const char which_dbm[] = "GNU gdbm"; /* EMX port of GDBM */
#elif defined(USE_GDBM_COMPAT)
#ifdef HAVE_GDBM_NDBM_H

In sum, this issue can be entirely avoided if the search ordering for dbm compatibility is reversed. If the environment provides gdbm with compat libraries, they should be fully preferred. I'm making a PR to this effect.

ambv added a commit to ambv/cpython that referenced this issue May 3, 2022
This makes macOS gdbm provided by Homebrew not segfault through correct
selection of the linked library (-lgdbm_compat) *AND* the correct ndbm-style
header (gdbm-ndbm.h instead of the invalid ndbm.h).
@ambv ambv linked a pull request May 3, 2022 that will close this issue
@ambv ambv removed a link to a pull request May 3, 2022
@ambv
Copy link
Contributor

ambv commented May 3, 2022

I unlinked the issue because this is not what we want: "linked a pull request that will close this issue"

We want to backport the fix.

ambv added a commit that referenced this issue May 3, 2022
This makes macOS gdbm provided by Homebrew not segfault through correct
selection of the linked library (-lgdbm_compat) *AND* the correct ndbm-style
header (gdbm-ndbm.h instead of the invalid ndbm.h).
@ambv
Copy link
Contributor

ambv commented May 3, 2022

See PR, we actually don't want to backport this. So this is fixed for 3.11+. For older Python versions, the workaround is to build with:

./configure --with-dbmliborder=gdbm:ndbm

@ambv ambv closed this as completed May 3, 2022
encukou added a commit to encukou/cpython that referenced this issue Oct 8, 2024
…th homebrew's gdbm 1.24

Per python#89452 (comment),
the issue is fixed in configure for 3.11+, and

> For older Python versions, the workaround is to build with:
>
>     ./configure --with-dbmliborder=gdbm:ndbm

We need this workaround in GitHub Actions, otherwise the tests fail.
@encukou
Copy link
Member

encukou commented Oct 8, 2024

This is now blocking 3.10 & 3.9 backport PRs. I'll re-open.

@encukou encukou reopened this Oct 8, 2024
ambv pushed a commit that referenced this issue Oct 8, 2024
…ebrew's gdbm 1.24 (#125112)

Per #89452 (comment),
the issue is fixed in configure for 3.11+, and

> For older Python versions, the workaround is to build with:
>
>     ./configure --with-dbmliborder=gdbm:ndbm

We need this workaround in GitHub Actions, otherwise the tests fail.
encukou added a commit to encukou/cpython that referenced this issue Oct 9, 2024
…h homebrew's gdbm 1.24 (pythonGH-125112)

Per https://github.com/python/cpython/issues/89452GH-issuecomment-1116329316,
the issue is fixed in configure for 3.11+, and

> For older Python versions, the workaround is to build with:
>
>     ./configure --with-dbmliborder=gdbm:ndbm

We need this workaround in GitHub Actions, otherwise the tests fail.
(cherry picked from commit 850189a)
ambv added a commit that referenced this issue Oct 9, 2024
…brew's gdbm 1.24 (GH-125112) (#125176)

Per https://github.com/python/cpython/issues/89452GH-issuecomment-1116329316,
the issue is fixed in configure for 3.11+, and

> For older Python versions, the workaround is to build with:
>
>     ./configure --with-dbmliborder=gdbm:ndbm

We need this workaround in GitHub Actions, otherwise the tests fail.
(cherry picked from commit 850189a)

Co-authored-by: Łukasz Langa <lukasz@langa.pl>
@ambv
Copy link
Contributor

ambv commented Oct 9, 2024

Thanks! Fixed and merged.

@ambv ambv closed this as completed Oct 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.9 only security fixes 3.10 only security fixes 3.11 only security fixes OS-mac type-crash A hard crash of the interpreter, possibly with a core dump
Projects
None yet
Development

No branches or pull requests

3 participants