SQ6 webserver occasionally crashes

[radio-engineer]

Companion 3.2.2, interfacing with an Allen & Heath SQ6. I saw the Caution flag saying the module hasn't been updated yet for version 3, but I tried it anyway.

The SQ6 webserver crashes seemingly randomly when the SQ module is active and connected. I didn't document exactly what it said last time it crashed, but it was something about a library sync. I'll test further and record the error message to update this thread. When the crash occurs, all SQ MixPad connections also die, and the only way to resolve it is to reboot the SQ6. That's the only issue I've found so far, but it's a showstopper when it breaks.

[JoshDarnIt-All]

Any update on exact error that is brought up on the crash?

Also if you run into it again grabbing the Log file of the instance would be helpful!

[radio-engineer]

@JoshDarnIt-All - so far haven't been able to find what consistenly crashes it, but I was able to pull a bunch of logs. There are also two different crash types we've observed. What we'll call the "Total Crash" causes the webserver on the SQ6 to crash and require a console reboot. The "Partial Crash" leaves the webserver functional with the MixPad app, but Companion glitches until you Quit and relaunch (Mac OS Silicon). We updated to 3.3.0 last week, tried 3.3.1 this morning, and ended up rolling back to 3.3.0 since the SQ6 wouldn't connect at all to 3.3.1.

Here are some debugs:

Partial Crash
system: ** Starting Connection from "/Applications/Companion.app/Contents/Resources/module-legacy/manifests/companion-module-allenheath-sq/index.js" **
system: ** Connection started **
Starting up module class: G
Sentry disabled
Module-host accepted registration
error: Failed to init: Error: Call timed out Error: Call timed out at t.IpcWrapper.sendWithCb (/Applications/Companion.app/Contents/Resources/main.js:2:2780215)
at x.init (/Applications/Companion.app/Contents/Resources/main.js:2:3213106)
at l.s (/Applications/Companion.app/Contents/Resources/main.js:2:3224562)
at l.emit (node:events:517:28)
at l.emit (node:domain:489:12)
at ChildProcess. (/Applications/Companion.app/Contents/Resources/main.js:2:2767212)
at ChildProcess.emit (node:events:517:28)
at ChildProcess.emit (node:domain:489:12)
at emit (node:internal/child_process:944:14)
at process.processTicksAndRejections (node:internal/process/task_queues:83:21)
system: ** Connection stopped **
error : MIDI error: write EPIPE
[sometimes it would spit out a TON of these MIDI errors]

Once the partial crash has happened, the only way to resolve it is to Quit and Re-launch the Companion service. The only sort-of consistency we found was certain faders interacting with -inf dB and +10dB (minimum and maximum on the fader scale) would bug out until we stopped the connection. Restarting the connection timed out until we relaunched Companion

===================================================

Full Crash
debug: Fader Received : 176,99,79,176,98,39,176,6,118,176,38,92 from 192.168.1.61
[lots of these "debug: Fader Received" messages (hundreds)]
debug: destroyed Gi5sJyDr4K5PkAcPcYr6l
error: node:events:495
throw er; // Unhandled 'error' event ^ Error [ERR_STREAM_DESTROYED]: Cannot call write after a stream was destroyed
at new NodeError (node:internal/errors:405:5)
at errorBuffer (node:internal/streams/writable:520:31)
at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:87:12) Emitted 'error' event on s instance at:
at /Applications/Companion.app/Contents/Resources/module-legacy/manifests/companion-module-allenheath-sq/index.js:1:132849
at errorBuffer (node:internal/streams/writable:520:5)
at afterWrite (node:internal/streams/writable:504:5)
at onwrite (node:internal/streams/writable:480:7)
at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:87:12) { code: 'ERR_STREAM_DESTROYED' } Node.js v18.20.2
system: ** Connection stopped **
has context menu

===================================================

The SQ MixPad threw two different errors during the total crash:
Failed to connect
Security sync timed out
AND
Failed to connect
Socket sync timed out
Not sure if those are helpful or not..

Hopefully that helps. If I can grab any additional info, let me know. I also have more duplicates of the log messages, but those two blocks are the groups issued during runtime surrounding the crashes. Also, if you need the logs in different form, let me know and I can upload txt files.

[JoshDarnIt-All]

Great info here! Thank you @radio-engineer
Running into the same partial crash every once in a while. I have an updated

I had issues with consistent "full Crashes" Until I updated the network infrastructure and moved everything to its own VLAN on static IPs. Since that move, I haven't run into a full lockup of the console. Could be helpful... also could not be helpful.

@josephdadams - This could be some helpful info to look through.

I have Tuesday, June 18th set aside for further testing of the module.

[RBBCUSER]

Any findings @JoshDarnIt-All ?

[JoshDarnIt-All]

Still no crashes on most recent release (3.3.1) But I have lost most control via the module with this release.
Will post issues in another thread.