Users cannot see other drawings

[jaysonl]

When using cocreate with a group of around 10 people two of them reported not being able to see any of the other's drawings. They could see each other's and we could see their annotations (including drawing over some of ours). They reported trying to switch browsers and still having an issue, but I failed to collect browser and OS information.

[edemaine]

Unfortunately, this isn't enough information to track anything down. What date/time did this happen? Browser information would also be helpful.

[jaysonl]

This occurred between 10:30am and noon ET on 9/28/2020. The specific events when we noticed and tested were very roughly around 11am, but the problem was never resolved during the meeting. I'll try to find a good way to ask about browser info.

Unrelatedly, we seemed to be experiencing significant lag around 11:50am.

[edemaine]

Weird. Server-side I only see around a brief 4-second delay at 11:50am, and don't see anything that should have caused anything after 10:30am. Maybe their network connection briefly disconnected, and that triggered some kind of bad state? I would have expected reconnection to work just fine, but haven't tested thoroughly.

I also restarted the servers in case there was something server-side, but I'm afraid I won't be able to figure this out. Thanks for the report, though; it's good to know something bad is happening. Maybe with more reports I'll eventually find the problem.

[edemaine]

FYI, I've now seen this affect another person, and I got to see a screenshare to see what the behavior was like. Conclusions:

• User-level workaround: the unfortunate user can reload the Cocreate page to get updates (provided they don't have another tab opened to the same Cocreate page), but it will continue to fail to update live. 🙁
• System-level workaround: restarting all the Meteor servers fixes the problem. But only the sysadmin can do this. 🙁 But let me know if you need me to do this again.
• I believe the issue is the server losing connection to Redis, reconnecting, but (maybe because of a race condition) some of Redis Oplog's subscriptions failing to be in a correct state. See redis in write-only/unsubscribed state after reconnect? cult-of-coders/redis-oplog#365 for my analysis so far.

[Alex7Li]

This has also happened to me a few times, my professor has lectures with about 8 people on the cocreate and about 1/4th of the time, I can't see edits until I refresh. Though I think it happens to me more frequently than other people. I tried clearing cookies and changing the browser from chrome to firefox, it did not change the behavior.

People can see edits that I make, though.

[Etn40ff]

I experienced the same issue yesterday. The weird part of it was that I could see my edits live on a different device, my collaborator could see mine and his edits, again on a separate device, but I could not see his.

[edemaine]

@Etn40ff That's expected behavior, I'm afraid. What's happening is that Cocreate "partitions" into multiple (between 2 and 4) "subnetworks", each of which properly updates within itself, but updates between the subnetworks don't propagate (without manual refresh). I still don't know how it happens, so don't know how to permanently fix it. The only fix I've found so far is to restart the Cocreate servers. 🙁

[Etn40ff]

This sounds quite bad but at least it is an explanation.

Currently, since I use it only with another person, I am piggybacking on the instance you run at MIT, I might have to look into self hosting if this happens again.

Thank you for the explanation (and for the software!)