Infinite recursion stack overflow when opening file dialog in MelonDS

[nyanpasu64]

When I launch MelonDS using AbsoluteTouchEx x64/atloader.exe, opening the file dialog makes the program crash.

Debugging

• I built AbsoluteTouchEx master 63244df myself, using the debug x64 configuration.
• I launched MelonDS using the debug binary. atloader injected into MelonDS and closed.
• I attached Visual Studio to MelonDS.
• Then I opened the file dialog to trigger the crash.

 	atdll.dll!std::_Compressed_pair<std::hash<HWND__ *>,std::_Compressed_pair<std::equal_to<HWND__ *>,float,1>,1>::_Get_first() Line 1339	C++
 	atdll.dll!std::_Uhash_compare<HWND__ *,std::hash<HWND__ *>,std::equal_to<HWND__ *>>::operator()<HWND__ *>(HWND__ * const & _Keyval) Line 174	C++
 	atdll.dll!std::unordered_map<HWND__ *,__int64 (__cdecl*)(HWND__ *,unsigned int,unsigned __int64,__int64),std::hash<HWND__ *>,std::equal_to<HWND__ *>,std::allocator<std::pair<HWND__ * const,__int64 (__cdecl*)(HWND__ *,unsigned int,unsigned __int64,__int64)>>>::at(HWND__ * const & _Keyval) Line 405	C++
>	atdll.dll!AT_WndProcHook(HWND__ * hWnd, unsigned int message, unsigned __int64 wParam, __int64 lParam) Line 927	C++
 	user32.dll!UserCallWinProcCheckWow()	Unknown
 	user32.dll!CallWindowProcW()	Unknown
 	duser.dll!ExtraInfoWndProc()	Unknown
 	user32.dll!UserCallWinProcCheckWow()	Unknown
 	user32.dll!CallWindowProcW()	Unknown
 	atdll.dll!AT_WndProcHook(HWND__ * hWnd, unsigned int message, unsigned __int64 wParam, __int64 lParam) Line 928	C++
 	user32.dll!UserCallWinProcCheckWow()	Unknown
 	user32.dll!CallWindowProcW()	Unknown
 	duser.dll!ExtraInfoWndProc()	Unknown
 	user32.dll!UserCallWinProcCheckWow()	Unknown
 	user32.dll!CallWindowProcW()	Unknown
...snipped...
 	atdll.dll!AT_WndProcHook(HWND__ * hWnd, unsigned int message, unsigned __int64 wParam, __int64 lParam) Line 928	C++
 	user32.dll!UserCallWinProcCheckWow()	Unknown
 	user32.dll!CallWindowProcW()	Unknown
 	duser.dll!ExtraInfoWndProc()	Unknown
 	user32.dll!UserCallWinProcCheckWow()	Unknown
 	The maximum number of stack frames supported by Visual Studio has been exceeded.	

I recall message was 124 or something, definitely not 0xFF or above.

I have several Windows Explorer addons installed though. I'm not sure if this can be reproduced on stock Windows 10.

melonDS 0.8.3 JIT beta for x64 Windows:

• http://melonds.kuribo64.net/comments.php?id=107
• http://kuribo64.net/get.php?id=nht0H4wC4lHhSUSZ

Windows 10 1909 x64.

[apsun]

Hmm interesting, one way I can imagine it got into this state is that the program calls the ANSI version of some WinAPI functions but the Unicode version of others. Currently AbsoluteTouchEx only hooks the Unicode versions. Let me see how hard it would be to support both.