code = 1006 (connection closed abnormally [internal]), no reason)

[oliver-zehentleitner]

What does this error mean?

https://tools.ietf.org/html/rfc6455 (search for 1006)
https://websockets.readthedocs.io/en/stable/faq.html#what-does-connectionclosederror-code-1006-mean

This topic at dev.binance.vision #1, #2, #3!

It depends from the power and quality of your CPU, internet connection, your OS, switches, loadbalancers, your NIF, Binance servers, ....

On a laptop via LTE I have a lot of reconnects with 1006. In a datacenter on a server I have very less of them.

Even the setup of the streams are relevant. Don't mix up to much different channel types. Make them dedicated, one channel like kline_1m and up to 1024 markets.

This lib is using multi-threading which is not really parallel in python - through the GIL all threads are executed sequential in a cycle. Divide 100 with your cpu cores and you have the max % of cpu power that your program (and this lib) can use. If your streams are needing more power than they can get, they restart with a 1006 error after a while!

Best regards,
Oliver

[oliver-zehentleitner]

Hi, not sure if this is the best place for this question, but I've noticed that before the 1006 error code comes, the stream has already been dead for several seconds. Is there some kind of parameter I could change at the code to force a restart before the 1006 error comes up and finally attempt to restart the stream?

In my case, with the streams I subscribe to, usually anything over 10 seconds without data means that the stream is dead.

Thanks!

@slipp1
like a "dead man's switch"? yes that’s easy possible, we just need to compare if the last heartbeat_timestamp < (time - timeout_time) and set a restart request.

but as long as they did not restart, its possible they heal also. if you drop your internet cable and put it back to get online again, the websockets connect again through the old established connection and the missing data gets flushed to your system.

may i ask you, how much data are you streaming in average and whats the power of your system and how much streams do you create? i think the best solution is to provide enough resources to avoid the 1006 error as much as possible.

[slipp1]

Thanks for the reply, I had deleted the comment cause it turned out it was pretty straightforward to do exactly how you said "last heartbeat_timestamp < (time - timeout_time) and set a restart request".

If you could give me some guidance here, what is your suggestion for a good restart request? I've tried:

1. stop_stream -> wait_till_stream_has_stopped -> create_stream
2. stop_stream -> sleep (1s) -> create_stream
   Both seem to work without any problems.

My 1006 codes are usually Binance server related, I have Futures bookTicker@arr and markPrice@arr only. CPU usage is about 10-20% of a vCPU, but it spikes up to 2-3x when the market moves a lot.

PS. Is there a difference between "stop_stream" and "kill_stream"?

[oliver-zehentleitner]

If I remember right, all you have to do is to use manager.set_restart_request(stream_id), this will trigger this code: https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/blob/master/unicorn_binance_websocket_api/unicorn_binance_websocket_api_manager.py#L592

Everything else should be done by the lib!

[oliver-zehentleitner]

I added ping_interval, ping_timeout, close_timeout to manager.create_stream()
and replace_stream()

[amaioli]

Hi Oliver, I face 1006 problem quite frequently today. The issue happens on different BOT installed on differents VM in Tokyo google DC at the same time so I don't think this time is related to ip connection but to an outage on Binance servers. In that case how long does Unicorn takes to automatically re-connect? Do you think that reducing the ping interval can minimize the connection downtime?

[oliver-zehentleitner]

could also be an outage of google DC or the wire between...

Every X seconds is a ping test, if it fails (after timeout) it disconnects and reconnects.

Also interesting: for you https://oliver-zehentleitner.github.io/unicorn-binance-websocket-api/unicorn_binance_websocket_api.html?highlight=restart_timeout#module-unicorn_binance_websocket_api.unicorn_binance_websocket_api_manager

If the Server has enough resources the streams are running very stable in general. I would try to find and solve the root cause.

[amaioli]

Thanks Oliver, a further question, Are there any way to catch and handle this event? BinanceWebSocketApiManager.stream_is_crashing

[oliver-zehentleitner]

Your welcome!
Yes there is a way: stream_signals

[amaioli]

Thanks Oliver, any change to collect stream_signals into process_stream_data?

[oliver-zehentleitner]

not sure if this is good, but to discuss this better start a new issue or move to #160