Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[browser] RuntimeError: memory access out of bounds #96546

Closed
pavelsavara opened this issue Jan 5, 2024 · 24 comments
Closed

[browser] RuntimeError: memory access out of bounds #96546

pavelsavara opened this issue Jan 5, 2024 · 24 comments
Labels
arch-wasm WebAssembly architecture area-Codegen-AOT-mono Known Build Error Use this to report build issues in the .NET Helix tab os-browser Browser variant of arch-wasm
Milestone

Comments

@pavelsavara
Copy link
Member

pavelsavara commented Jan 5, 2024

Error Blob

{
  "ErrorMessage": "RuntimeError: memory access out of bounds",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=515167
Log

[20:26:27] warn: worker sent an error! http://127.0.0.1:39745/_framework/dotnet.native.worker.js:1: Uncaught RuntimeError: memory access out of bounds
[20:26:27] fail: MONO_WASM: Error
                     at Ze (http://127.0.0.1:39745/_framework/dotnet.js:3:19735)
                     at e (http://127.0.0.1:39745/_framework/dotnet.js:3:33729)
                     at http://127.0.0.1:39745/_framework/dotnet.js:3:33874
[20:26:27] warn: MONO_WASM: mono_wasm_exit failed unwind

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=515167
Error message validated: RuntimeError: memory access out of bounds
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 1/5/2024 12:55:21 PM UTC

Report

Build Definition Test Pull Request
774241 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106329
774196 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106317
773979 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106013
773665 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106166
773561 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106169
773348 dotnet/runtime WasmTestOnV8-ST-System.Text.Encodings.Web.Tests.WorkItemExecution
773213 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106166
773113 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution
772876 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106271
772314 dotnet/runtime WasmTestOnChrome-ST-System.Text.Encodings.Web.Tests.WorkItemExecution
772028 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106037
771854 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106226
771814 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution
771612 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106169
771470 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106208
770671 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution
770319 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106164
770038 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106153
770007 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106150
769899 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106145
769813 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106111
769640 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105941
769674 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105559
769572 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105866
769545 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106123
769451 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106111
769299 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106108
769093 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105663
768949 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106091
768676 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #104393
768667 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106053
768671 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106080
763399 dotnet/runtime WasmTestOnChrome-ST-System.Text.Encodings.Web.Tests.WorkItemExecution
768349 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #99596
768211 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106062
768078 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106030
767917 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105841
767877 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #106048
767709 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution
767356 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105841
766122 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105941
765139 dotnet/runtime WasmTestOnChrome-ST-System.Text.Encoding.Tests.WorkItemExecution
764961 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105910
764868 dotnet/runtime WasmTestOnV8-ST-System.Threading.Tasks.Dataflow.Tests.WorkItemExecution
764572 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105846
764553 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution
763838 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105548
763775 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution
763718 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105814
763499 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105782
763419 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105776
763061 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105548
762675 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105764
762167 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105765
762078 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105709
761851 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105765
761368 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105464
761344 dotnet/runtime WasmTestOnV8-ST-System.Text.Encoding.Tests.WorkItemExecution
761211 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105654
760839 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105709
760586 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105709
760243 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105471
760209 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105691
760113 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105561
759876 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105579
759555 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution
759524 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105424
759413 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105643
759407 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105642
759349 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105636
758737 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105559
758686 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105489
758465 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105610
758407 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #104849
758343 dotnet/runtime WasmTestOnChrome-ST-System.Text.Encodings.Web.Tests.WorkItemExecution
758133 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105471
758115 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105469
757874 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution
757596 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #105548

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 79
@pavelsavara pavelsavara added arch-wasm WebAssembly architecture blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' area-VM-threading-mono Known Build Error Use this to report build issues in the .NET Helix tab os-browser Browser variant of arch-wasm labels Jan 5, 2024
@pavelsavara pavelsavara added this to the 9.0.0 milestone Jan 5, 2024
@pavelsavara pavelsavara self-assigned this Jan 5, 2024
@ghost
Copy link

ghost commented Jan 5, 2024

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Error Blob

{
  "ErrorMessage": "RuntimeError: memory access out of bounds",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=515167
Log

[20:26:27] warn: worker sent an error! http://127.0.0.1:39745/_framework/dotnet.native.worker.js:1: Uncaught RuntimeError: memory access out of bounds
[20:26:27] fail: MONO_WASM: Error
                     at Ze (http://127.0.0.1:39745/_framework/dotnet.js:3:19735)
                     at e (http://127.0.0.1:39745/_framework/dotnet.js:3:33729)
                     at http://127.0.0.1:39745/_framework/dotnet.js:3:33874
[20:26:27] warn: MONO_WASM: mono_wasm_exit failed unwind```

<table>
  <tr>
    <th align="left">Author:</th>
    <td>pavelsavara</td>
  </tr>
  <tr>
    <th align="left">Assignees:</th>
    <td>pavelsavara</td>
  </tr>
  <tr>
    <th align="left">Labels:</th>
    <td>

`arch-wasm`, `blocking-clean-ci`, `area-VM-threading-mono`, `Known Build Error`, `os-browser`

</td>
  </tr>
  <tr>
    <th align="left">Milestone:</th>
    <td>9.0.0</td>
  </tr>
</table>
</details>

@pavelsavara
Copy link
Member Author

pavelsavara commented Jan 11, 2024

Another
Build
Log

[23:11:59] info: [2024-01-10T23:11:59.397Z] [PASS] System.Threading.ThreadPools.Tests.ThreadPoolTests.RunProcessorCountItemsInParallel
[23:11:59] info: [2024-01-10T23:11:59.400Z] [SKIP] System.Threading.ThreadPools.Tests.ThreadPoolTests.SetMinMaxThreadsTest_ChangedInDotNetCore
[23:12:00] warn: worker sent an error! http://127.0.0.1:33633/_framework/dotnet.native.worker.js:1: Uncaught RuntimeError: memory access out of bounds
[23:12:00] fail: MONO_WASM: Error
                     at Ze (http://127.0.0.1:33633/_framework/dotnet.js:3:19735)
                     at e (http://127.0.0.1:33633/_framework/dotnet.js:3:33729)
                     at http://127.0.0.1:33633/_framework/dotnet.js:3:33874
[23:12:00] info: WASM EXIT 1
[23:12:00] info: Waiting to flush log messages with a timeout of 120 secs ..

@pavelsavara
Copy link
Member Author

Another Log
Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=549038&view=logs&jobId=63c2d0c8-fec2-5788-81c8-f3ac95e8841f

[11:44:47] fail: [0x066a4300-long 11:44:47.111] MONO_WASM: Unhandled error: {}
[11:44:47] fail: [0x066a4300-long 11:44:47.117] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[8732]:0x217b29
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[2731]:0xd2e2b
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[2815]:0xd597a
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[2077]:0xaed21
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[324]:0x286c7
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[253]:0x25e62
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[244]:0x18884
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[283]:0x26af0
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[3319]:0xf9db8
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[2587]:0xcbe17
[11:44:47] info: WASM EXIT 1

@pavelsavara
Copy link
Member Author

Log

[08:55:30] info: Starting:    Microsoft.Extensions.Logging.Console.Tests.dll
[08:55:31] fail: [0x045f6788-emsc 08:55:31.650] MONO_WASM: preRunWorker() failed {}
[08:55:31] fail: [0x045f6788-emsc 08:55:31.652] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at http://127.0.0.1:44985/_framework/dotnet.native.wasm:wasm-function[649]:0x4a787
                     at http://127.0.0.1:44985/_framework/dotnet.native.wasm:wasm-function[650]:0x4a7d6
                     at http://127.0.0.1:44985/_framework/dotnet.native.wasm:wasm-function[652]:0x4a8ab
                     at http://127.0.0.1:44985/_framework/dotnet.native.wasm:wasm-function[530]:0x4793a
                     at e.<computed> (http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:226093)
                     at http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:101299
                     at Ga (http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:101426)
                     at Ya (http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:101713)
                     at http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:212904
                     at iu (http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:213061)
[08:55:31] info: [0x000dc84c-main 08:55:31.655] MONO_WASM: Dumping web worker info as seen by UI thread, it could be stale:
[08:55:31] info: 00 | 0x000dc84c-main: isRunning:false isAttached: true isEventLoop:false reuseCount:  1 - UI Thread
[08:55:31] info: 01 | 0x02ff8f90-norm: isRunning: true isAttached: true isEventLoop:false reuseCount:  1 - Finalizer
[08:55:31] info: 02 | 0x032a8030-pool: isRunning: true isAttached: true isEventLoop: true reuseCount:  1 - .NET TP Worker
[08:55:31] info: 03 | 0x035b0040-gate: isRunning: true isAttached: true isEventLoop:false reuseCount:  1 - .NET TP Gate
[08:55:31] info: 04 | 0x03dbea48-pool: isRunning: true isAttached: true isEventLoop: true reuseCount:  1 - .NET TP Worker
[08:55:31] info: 05 | 0x045f6788-emsc: isRunning: true isAttached:false isEventLoop:false reuseCount:  1 - pthread-assigned
[08:55:31] info: 06 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 07 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 08 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 09 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 10 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 11 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 12 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] fail: [0x000dc84c-main 08:55:31.657] MONO_WASM: Uncaught RuntimeError: memory access out of bounds
                 Error
                     at Object.Ge [as mono_exit] (http://127.0.0.1:44985/_framework/dotnet.js:3:20140)
                     at ft.config.exitOnUnhandledError.e.onerror (http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:38634)
[08:55:31] fail: [out of order message from the browser]: http://127.0.0.1:44985/_framework/dotnet.native.js 7:1295 Uncaught ErrorEvent: Uncaught RuntimeError: memory access out of bounds

@pavelsavara
Copy link
Member Author

During thread creation
Log
Build

[18:39:28] fail: [--06290040-emsc 18:39:28.771] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at pthread_attr_init (http://127.0.0.1:34537/_framework/dotnet.native.wasm:wasm-function[8640]:0x2144c1)
                     at mono_thread_info_attach (http://127.0.0.1:34537/_framework/dotnet.native.wasm:wasm-function[786]:0x4ef10)
                     at start_wrapper (http://127.0.0.1:34537/_framework/dotnet.native.wasm:wasm-function[2762]:0xd6903)
                     at Object.invokeEntryPoint (http://127.0.0.1:34537/_framework/dotnet.native.js:8:69009)
                     at handleMessage (http://127.0.0.1:34537/_framework/dotnet.native.worker.js:1:2306)

@pavelsavara
Copy link
Member Author

Related #98630

@pavelsavara
Copy link
Member Author

Log

[11:23:26] info: 001 | 0x03078030-pool: isRunning:false isAttached: true isEventLoop: true reuseCount:  1 - .NET TP Worker
[11:23:26] fail: [0x03078030-pool 11:23:26.696] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at dlmalloc (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[8788]:0x21d1e6)
                     at dlcalloc (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[8796]:0x21f96a)
                     at monoeg_g_calloc (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[598]:0x49521)
                     at monoeg_malloc0 (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[599]:0x49544)
                     at monoeg_g_array_new (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[569]:0x48b40)
                     at mono_interp_transform_method (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[438]:0x407b6)
                     at do_transform_method (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[255]:0x25481)
                     at mono_interp_exec_method (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[254]:0x251fb)
                     at interp_entry (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[327]:0x28391)
                     at interp_entry_static_2 (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[365]:0x29411)

@lewing
Copy link
Member

lewing commented Feb 21, 2024

cc @vargaz @steveisok

@pavelsavara
Copy link
Member Author

local run with nice stack trace

fail: [0x05a6fef0-jsww 15:29:09.596] MONO_WASM: memory access out of bounds
  RuntimeError: memory access out of bounds
      at dlfree (dlfree (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[17333]:0x34eda4))
      at bound_fn (http://127.0.0.1:61068/_framework/dotnet.runtime.js:4146:24)
      at sync_bound_fn (http://127.0.0.1:61068/_framework/dotnet.runtime.js:3980:13)
      at JSImport_INTERNAL_http_wasm_supports_streaming_request (https://dotnet/JSImport/INTERNAL_http_wasm_supports_streaming_request:4:86)
      at mono_wasm_invoke_jsimport (http://127.0.0.1:61068/_framework/dotnet.runtime.js:3893:5)
      at do_icall (do_icall (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[231]:0x29de6))
      at do_icall_wrapper (do_icall_wrapper (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[151]:0x26971))
      at mono_interp_exec_method (mono_interp_exec_method (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[142]:0x150ac))
      at interp_runtime_invoke (interp_runtime_invoke (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[185]:0x27d50))
      at mono_jit_runtime_invoke (mono_jit_runtime_invoke (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[9174]:0x200a97))

@pavelsavara
Copy link
Member Author

another local with great log

MONO_WASM: memory access out of bounds
RuntimeError: memory access out of bounds
    at copy_object_no_checks.1 (copy_object_no_checks.1 (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[2325]:0xa15c1))
    at simple_nursery_serial_scan_object (simple_nursery_serial_scan_object (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[2321]:0x9f935))
    at simple_nursery_serial_drain_gray_stack (simple_nursery_serial_drain_gray_stack (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[2324]:0xa1454))
    at sgen_drain_gray_stack (sgen_drain_gray_stack (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1721]:0x7c5ca))
    at finish_gray_stack (finish_gray_stack (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1828]:0x812be))
    at collect_nursery (collect_nursery (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1744]:0x7d6ee))
    at sgen_perform_collection_inner (sgen_perform_collection_inner (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1741]:0x7d11f))
    at sgen_perform_collection (sgen_perform_collection (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1740]:0x7d001))
    at sgen_ensure_free_space (sgen_ensure_free_space (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1739]:0x7cfac))
    at sgen_alloc_obj_nolock (sgen_alloc_obj_nolock (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1669]:0x7af3e))```

@pavelsavara
Copy link
Member Author

another local

RuntimeError: memory access out of bounds
    at dlfree (dlfree (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[17410]:0x34e69e))
    at bound_fn (https://localhost:65453/_framework/dotnet.runtime.js:4217:24)
    at async_bound_fn (https://localhost:65453/_framework/dotnet.runtime.js:4027:9)
    at JSImport_INTERNAL_dynamic_import (https://dotnet/JSImport/INTERNAL_dynamic_import:4:64)
    at mono_wasm_invoke_jsimport_MT (https://localhost:65453/_framework/dotnet.runtime.js:3948:5)
    at do_icall (do_icall (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[227]:0x29b34))
    at do_icall_wrapper (do_icall_wrapper (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[147]:0x266c6))
    at mono_interp_exec_method (mono_interp_exec_method (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[138]:0x150fd))
    at interp_runtime_invoke (interp_runtime_invoke (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[181]:0x27aa5))
    at mono_jit_runtime_invoke (mono_jit_runtime_invoke (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[9240]:0x2001ac))```

@pavelsavara
Copy link
Member Author

Most of the problems are fixed by #100304

There is new occurrence as fallout of OOM
Log

[19:33:04] info: [FAIL] System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers(options: System.Text.Json.JsonWriterOptions, keyString: "mess><age")
[19:33:04] info: System.OutOfMemoryException : Out of memory
[19:33:04] info:    at System.GC.AllocateArray[Char](Int32 length, Boolean pinned)
[19:33:04] info:    at System.Text.StringBuilder.ExpandByABlock(Int32 minBlockCharCount)
[19:33:04] info:    at System.Text.StringBuilder.AppendWithExpansion(Char value)
[19:33:04] info:    at System.Text.StringBuilder.Append(Char value)
[19:33:04] info:    at System.IO.StringWriter.Write(Char value)
[19:33:04] info:    at Newtonsoft.Json.JsonTextWriter.WriteEnd(JsonToken token)
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.AutoCompleteClose(JsonContainerType type)
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.WriteEndArray()
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.WriteEnd(JsonContainerType type)
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.WriteEnd()
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.AutoCompleteAll()
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.Close()
[19:33:04] info:    at Newtonsoft.Json.JsonTextWriter.Close()
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.Dispose(Boolean disposing)
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.System.IDisposable.Dispose()
[19:33:04] info:    at System.Text.Json.Tests.Utf8JsonWriterTests.GetExpectedLargeArrayOfStrings(Int32 length)
[19:33:04] info:    at System.Text.Json.Tests.Utf8JsonWriterTests.WriteJsonOnlyWritesToStreamOnDemand_FlushAsync()
[19:33:04] info: --- End of stack trace from previous location ---
[19:33:04] info:    at System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers(JsonWriterOptions options, String keyString)
[19:33:04] info:    at System.Object.InvokeStub_Utf8JsonWriterTests.WriteNumbers(Object , Span`1 )
[19:33:04] info:    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
[19:33:04] fail: [0x020cdfc0-dpty 19:33:04.706] MONO_WASM: Unhandled error: RuntimeError: memory access out of bounds
[19:33:04] fail: [0x020cdfc0-dpty 19:33:04.707] MONO_WASM: Unhandled error: RuntimeError: memory access out of bounds
[19:33:04] info: [0x020cdfc0-dpty 19:33:04.710] MONO_WASM: Dumping web worker info as seen by UI thread, it could be stale:
[19:33:04] info: 001 | 0x020cdfc0-dpty: isRunning: true isAttached: true isEventLoop:false reuseCount:  1 -
[19:33:04] fail: [0x020cdfc0-dpty 19:33:04.748] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at copy_object_no_checks.1 (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[1115]:0x637d7)
                     at simple_nursery_serial_scan_object (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[1113]:0x63003)
                     at simple_nursery_serial_drain_gray_stack (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[1110]:0x626d8)
                     at finish_gray_stack (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[955]:0x56d6a)
                     at collect_nursery (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[927]:0x5481b)
                     at sgen_perform_collection (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[925]:0x540da)
                     at sgen_ensure_free_space (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[924]:0x53f75)
                     at sgen_alloc_obj_nolock (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[911]:0x533bc)
                     at mono_gc_alloc_string (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[3174]:0xf0e73)
                     at mono_string_new_size_checked (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[2645]:0xd1efe)
                     at mono_interp_exec_method (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[257]:0x208b2)
                     at interp_entry (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[330]:0x286f9)
                     at interp_entry_static_0 (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[366]:0x29702)
                     at wasm_native_to_interp_System_Runtime_InteropServices_JavaScript_JSSynchronizationContext_BackgroundJobHandler (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[232]:0x1676e)
                     at mono_background_exec (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[836]:0x50e85)
                     at Os (http://127.0.0.1:35135/_framework/dotnet.runtime.js:3:69709)
                     at callUserCallback (http://127.0.0.1:35135/_framework/dotnet.native.js:8:102608)
                     at http://127.0.0.1:35135/_framework/dotnet.native.js:8:111163
[19:33:04] info: WASM EXIT 1

@kg
Copy link
Member

kg commented Apr 9, 2024

@pavelsavara
Copy link
Member Author

This incorrectly ate https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-100801-merge-8c5cd227f4844ef98c/Methodical_d2/1/console.e95f9cb2.log?helixlogtype=result from a non-MT lane, which looks like it could be a real issue.

I already opened #100757 for it

@lewing
Copy link
Member

lewing commented Apr 13, 2024

this appears to have been fixed with #100907 closing so that new failures are caught

@lewing lewing closed this as completed Apr 13, 2024
@lewing lewing reopened this Apr 18, 2024
@lewing
Copy link
Member

lewing commented Apr 18, 2024

hit again in #101179

@pavelsavara
Copy link
Member Author

pavelsavara commented Apr 18, 2024

There are 4 more this morning, 3 of them are after OOM.
The 4th is in GC (probably on the edge of OOM too)
https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-101122-merge-f513e5fd0d4a45d08e/WasmTestOnBrowser-System.Text.Json.Tests/1/console.93cb60d2.log?helixlogtype=result

Perhaps some alloc failed and returned NULL and we are not checking it ?

@pavelsavara pavelsavara removed their assignment Apr 18, 2024
@pavelsavara
Copy link
Member Author

pavelsavara commented Apr 18, 2024

Alternatively, are we missing some memory fence which would synchronize the memory between WASM threads ?

We do issue .grow(0) in mono_wasm_synchronization_context_pump but sgen_perform_collection could be doing lot of work and the memory could grow on another thread in the meantime.

[09:03:07] fail: [0x020ff9e8-dpty 09:03:07.808] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at copy_object_no_checks.1 (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[1119]:0x677c9)
                     at simple_nursery_serial_scan_object (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[1117]:0x66fe9)
                     at simple_nursery_serial_drain_gray_stack (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[1114]:0x666be)
                     at finish_gray_stack (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[957]:0x5a74b)
                     at collect_nursery (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[930]:0x5813f)
                     at sgen_perform_collection (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[928]:0x579f4)
                     at sgen_ensure_free_space (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[927]:0x5788d)
                     at sgen_alloc_obj_nolock (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[914]:0x56d40)
                     at mono_gc_alloc_vector (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[3167]:0xfab94)
                     at mono_array_new_specific_internal (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[2710]:0xddfeb)
                     at mono_array_new_specific_checked (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[2711]:0xde022)
                     at mono_interp_exec_method (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[257]:0x21463)
                     at interp_entry (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[330]:0x297e8)
                     at interp_entry_static_0 (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[366]:0x2a830)
                     at wasm_native_to_interp_System_Runtime_InteropServices_JavaScript_JSSynchronizationContext_PumpHandler (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[232]:0x171f1)
                     at mono_wasm_synchronization_context_pump (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[206]:0x1556e)

@kg
Copy link
Member

kg commented Apr 18, 2024

I think under normal circumstances we shouldn't need to .grow(0), the theory was that we specifically needed to do it if the debugger paused thread(s) while a grow operation happened, wasn't it?

Unchecked null pointer feels very plausible. It will fail silently and also corrupt the zero page (which will cause jiterp traces to break.)

@pavelsavara
Copy link
Member Author

I saw Log

[13:54:11] fail: Error in mono_download_assets: RangeError: Start offset -1 is outside the bounds of the buffer
[13:54:11] fail: [--000eafbc-emsc 13:54:11.333] MONO_WASM: Start offset -1 is outside the bounds of the buffer
                 RangeError: Start offset -1 is outside the bounds of the buffer
                     at new Uint8Array (<anonymous>)
                     at http://127.0.0.1:41625/_framework/dotnet.runtime.js:3:77638
                     at Object.pa [as instantiate_asset] (http://127.0.0.1:41625/_framework/dotnet.runtime.js:3:77686)
                     at n (http://127.0.0.1:41625/_framework/dotnet.js:3:8629)
                     at async Promise.all (index 129)
[13:54:11] fail: [out of order message from the browser]: http://127.0.0.1:41625/_framework/dotnet.runtime.js 2:77637 Uncaught RangeError: Start offset -1 is outside the bounds of the buffer
[13:54:11] info: WASM EXIT 1

This translates to ICU asset data being loaded into memory via mono_wasm_load_bytes_into_heap_persistent on MT build in UI thread, while the mono is starting on deputy thread.

It seems that Module._sbrk returned -1 🤯

It seems that sbrk could be racing with other threads and loose.
https://github.com/emscripten-core/emscripten/blob/1ca2f3141ec99110d6c5c5690dcd1c65d08d97e5/system/lib/libc/sbrk.c#L75

Also I'm not sure on what version of emscripten we are now W.R.T. https://github.com/emscripten-core/emscripten/pull/20793/files

We are just lucky it was called from JS and that we got nice JS exception.
I bet the same could happen to our C code as well.
Which would mean we could get RuntimeError: memory access out of bounds unless all our code is handling -1 and NULL returns from sbrk and malloc

@kg
Copy link
Member

kg commented Jul 17, 2024

It looks like this was originally intended to match MT failures, but based on the recent matches in the automated list, it's matching ST failures. That's not good, memory access out of bounds on ST is more likely to be real bugs that we introduced in PRs and now it will be green.

@pavelsavara
Copy link
Member Author

They are JS error, it will not capture the same issue if it happens on Windows target for example.

When access out of bounds happens in ST it only makes troubleshooting MT more difficult.

@lewing lewing changed the title [browser][MT] RuntimeError: memory access out of bounds [browser] RuntimeError: memory access out of bounds Aug 15, 2024
@lewing
Copy link
Member

lewing commented Aug 15, 2024

This has transitioned to ST only then dropped off after the alignment fix 🤞

@lewing lewing modified the milestones: 9.0.0, 10.0.0 Aug 15, 2024
@ilonatommy
Copy link
Member

Not being hit after alignment PR: #106313.

@github-actions github-actions bot locked and limited conversation to collaborators Sep 28, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
arch-wasm WebAssembly architecture area-Codegen-AOT-mono Known Build Error Use this to report build issues in the .NET Helix tab os-browser Browser variant of arch-wasm
Projects
None yet
Development

No branches or pull requests

5 participants