Windows: exporting a tuple leads to "A dynamic link library (DLL) initialization routine failed."

[paugier]

I discovered a funny bug affecting Transonic users on Windows when using Visual Studio and clang-cl (but not MinGW, which explains why I didn't detect that with our CI on Github Actions). Pythran files written by Transonic export a tuple of strings, which usually work fine but not for this specific case.

It is quite simple to reproduce with a Windows machine (details because working on Windows is new for me):

• install python 3.11 from the Microsoft Store
• install Visual Studio 2022 with clang-cl
• open "x64 native tools command prompt for VS 2022" (to target x64)

We get:

clang-cl --version
clang version 17.0.3
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\Llvm\x64\bin

python --version
Python 3.11.8

• create a venv, activate it and install pythran

python -m venv .venv
.venv\Scripts\activate.bat
pip install pythran

• create a file simple.py

#pythran export my_tuple
my_tuple = ("a",)

• Set the environment variables with

set CC=clang-cl
set CXX=clang-cl

• create the extension with pythran simple.py

• import the extension python -c "import simple" gives:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: DLL load failed while importing simple: A dynamic link library (DLL) initialization routine failed.

Note: actually one can also reproduce simply with an environment based on conda-forge created with mamba create -n env_pythran pythran clang (in this case clang==18.1.2).

Even though I guess exporting a tuple from a pythran file is not common, this bug is quite annoying for me because it affects all Pythran extensions produced through Transonic on Windows for conda-forge 🙂 Therefore, I think I'm going to quickly release a Transonic which does not export this tuple to overcome this bug.

[paugier]

Additional information about this bug:

• I cannot reproduce on Linux with conda-forge.
• transonic 0.6.4 avoids this bug for most cases 🙂
• this bug can also be triggered by exporting a dictionary 🙁 , which is much more annoying because it is used for some extensions by Transonic and it cannot be simply avoided. I'm going to give a minimum example next week when I have access to a Windows machine.

[serge-sans-paille]

It's difficult for me to debug that one as I don't have access to a windows machine. I confirm I can't reproduce on windows. Could you try the following:

pythran -E a.py
sed -i -e 's/PyModule_AddObject/PyModule_AddObjectRef/g' a.cpp
pythran a.cpp
python -c 'import a; print(a.my_tuple)'

I don't see why this would change the behavior (it should actually introduce a leak), just trying.

[paugier]

Replacing PyModule_AddObject by PyModule_AddObjectRef changes nothing.

If I comment PyModule_AddObject(theModule, "my_tuple", my_tuple);, same thing.

If I comment static PyObject* my_tuple = to_python(__pythran_simple::my_tuple()());, the extension can be imported (but of course without my_tuple).

Then, I tried to replace this line with

Py_ssize_t size = 2;
PyObject* my_tuple = PyTuple_New(size);

I again get the same error.

Then, I tried to manually write a minimal cpp file exporting one tuple and I again get the same error, but I really don't know if it is correct.

#include <Python.h>

Py_ssize_t size = 2;
PyObject* my_tuple = PyTuple_New(size);

static PyMethodDef methods[] = {
    {NULL, NULL, 0, NULL}  /* Sentinel */
};

static struct PyModuleDef mod_def = {
    PyModuleDef_HEAD_INIT,
    "my_mod",
    "",
    -1,
    methods,
    NULL,
    NULL,
    NULL,
    NULL
};

PyObject *PyInit_my_mod(void)
{
    PyObject* theModule = PyModule_Create(&mod_def);
    PyModule_AddObject(theModule, "my_tuple", my_tuple);
    return theModule;
}

I tried to check on Linux but using pythran to compile such minimal cpp file does not work on Linux ("ImportError: dynamic module does not define module export function (PyInit_my_mod)").

[char101]

I tried this and it seems the problem is that the tuple is created before PYTHRAN_MODULE_INIT is called.

If the variable is moved just before PyModule_AddObject then the module can be imported without problem.

static PyObject *my_tuple = to_python(__pythran_test::my_tuple()());
PyModule_AddObject(theModule, "my_tuple", my_tuple);

Python 3.13.0 (main, Oct  9 2024, 09:35:48) [Clang 19.1.0 ] 64 bit (AMD64) with MSC v.1939 CRT] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> test.my_tuple
('a',)

[serge-sans-paille]

Thanks @char101 ! #2269 should do the trick then, could you (and @paugier ) give it a try?

[char101]

Tested and works fine.

# pythran export a
a = 1
# pythran export b
b = ('2',)
# pythran export c
c = {3: True}

Python 3.13.0 (main, Oct  9 2024, 09:35:48) [Clang 19.1.0 ] 64 bit (AMD64) with MSC v.1939 CRT] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import test3
>>> test3.a
1
>>> test3.b
('2',)
>>> test3.c
{3: True}

[serge-sans-paille]

yay! thanks for the hint :-) btw, in which context are you interacting with pythran?

[char101]

btw, in which context are you interacting with pythran?

By context do you mean where is pythran used? test3 in this case is a .pyd already compiled with clang-cl using pythran.

Python 3.13.0 (main, Oct  9 2024, 09:35:48) [Clang 19.1.0 ] 64 bit (AMD64) with MSC v.1939 CRT] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import test3
>>> test3.__file__
'R:\\test3.cp313-win_amd64.pyd'
>>> import os
>>> os.listdir('.')
['$RECYCLE.BIN', 'System Volume Information', 'test3.cp313-win_amd64.pyd', 'test3.py']
>>> test3.a
1

[serge-sans-paille]

No, I meant, what would be the project you're working on that uses pythran, just being curious.

[char101]

My use case is nothing big really, just speeding up loops for technical indicators.

[serge-sans-paille]

Thanks for going all the way down to tracking this bug then ;-)