Persistence and microservices

[guillaumerosinosky]

Hello, first, great work ! I like a lot Scribble.rs, it's feature-rich, well-developed and with a very polished UI.

In my research team, we plan to use your project for our current public research on latency-sensitive distributed (edge/cloud) stateful microservices applications (in the experimentation part, to demonstrate our allocation methods work well on a representative application). Of course, we need a stateful microservices application for this, and your application being simple and good as it is, don't need a lot of work to be a good example to show. We have begun to enrich it first with persistence, and plan directly afterwards to add some relevant microservices.

For this, I have already added some basic Redis based persistence in my fork (https://github.com/guillaumerosinosky/scribble.rs): except a few bugs, it's possible to shutdown the server in-game, and reconnect to it (with a refresh of the browser). I also plan to add a few microservices (for now two: registered users/score history, and dictionnary).

Two questions/propositions:

• concerning the persistence, I can polish a bit and add some basic tests for a PR, if you think it can be an interesting for you. For now the whole state of the lobby is persisted for each event, and only loaded at init, but I plan to separate the persistence of the currentDrawing, and load it at other times. Could you give me your two cents on this?
• do you think other microservices could be useful (depending on the features you plan etc.) ? If it is interesting for us to show, I can add these on my fork, and they can be useful for the project as well.

Thank you :)

[Bios-Marcel]

Hey, my initial plan was not to have any persistence. The whole thing was supposed to be something easy to setup, run and not require any backing up.

Accounts are only really necessary in order to save stuff such as highscores. However, I think highscores are to a degree nonsensical, as each game has different words, people, roundcount, drawingtime and whatnot. So a score would not be representative for actual player performance. On top of that, the game was mostly intent for casual playing with friends. E.g. a social environment, instead of a competitive one.

However, I assume for the purpose of research, this is fine.

The only additional service that I planned making, was a wordlist server + frontend. Where people could add and edit word lists. For that, accounts might be useful in the future, but currently I have no real usecases for the main server.

The only usecase that I could think of for the persistence layer would be server updates. This way I could update a running instance without worrying about having to ruin a running game. This would probs require some addiotional effort though, as reconnecting might get more complicated due to frontend changes introduced by the update. I am not sure whether that's worth tho.

However, feel free to share your research progress here.

If you need any help or have questions, I'll gladly help a bit.

[guillaumerosinosky]

Thank you for your answer.

About the score, I understand the variety problem, and your "social" target. I guess what we'll assume for our fork is this service could permit users to consult their past games and scores. High scores/rankings should matter only on "official" competition games with a fixed configuration.

About the word list (sorry used the term "dictionnary" but I meant it), I see. I'll do a basic version of it as well, most probably during the next week.

Another point for persistence: it's a way to have scalability/elasticity and high availability features. For instance if you have hundreds/thousands of lobbies, one server could not be enough and you may want to scale out. Maybe you could want to reduce the number of servers as well when there is no activity. If there is a crash in the datacenter/VM hosting the server, the current games will be lost. But I understand this is not the point of your project, though.

About the server upgrade problematic: as long as the database structure stays the same (or is compatible), I guess it could be possible to force a reload in the browser (or ask a user to do it). Even with the current (basic) persistence behaviour I added, it works. The only issue I had is about the timeLeftPicker, but when recreating it when loading the lobby from the database, it seems to work (knowing that during unavailability, time "continues to flow", questionable I guess).

I'll keep you in touch with our progress. We may at some point in time prepare some Kubernetes Helm chart/operator deployment files as well. Of course, since our work is on backend orchestration, we probably won't do the front-end part for the new services.

[Bios-Marcel]

Well, i don't know a lot about scalability and stuff so I don't know how easy it is or what the drawbacks are and whatnot. While it would be cool, it's currently not relevant I guess, since this is more of a small self hosted thing. Not many people use the official instance and I guess that's good. I currently host it on a weak free heroku dyno.

[guillaumerosinosky]

I understand. If in the future you need scalability and/or high availability, this fork will be there :).

Another point we'll do: some load testing/latency+throughput measurements/distributed tracing (https://opentelemetry.io/). Not sure yet what we will use though as a loadtest library, since we need it to support WebSocket and realistic fixed throughput scenarios. Will tell you.

[Bios-Marcel]

Btw, a couple of events do not cause state changes. You might not want to save the state on each event. This also represents an abuse potential. Not sure whether that's relevant for you though.

[Bios-Marcel]

Of course the drawback of this approach is that it can take time (and lots of events on the network) to replay a long game

Well, for my usecase I was really thinking about something that feels like a recorded video. So it'd take as long as the game took.
However, for your usecase you won't have any heavy network traffic, since you replay everything locally on server. During the replay, you can omit all eventing, as there's no player "listening". Now when a player refreshes / reconnects, they simply connect like they would usually do, receiving only a single ready event.

[Bios-Marcel]

I don't know whether I'll do this API yet, better forget about it for now, I don't want to do empty promises here.
I guess there's more important stuff to do. I am mostly fixing small bugs, improving code and docs right now.
I also have some more problematic bugs I should fix.

I recommend you merge changes from time to time ^^

[guillaumerosinosky]

Ok, I see about the replay. You already have something that manages this (did not see it before).

I'll synchronize yes. FYI, pushed this afternoon some more profound modifications to manage OpenTelemetry distributed traces on my fork. Simply added traceId and spanId to the payload of GameEvent, LineEvent, FillEvent. I had to add context (https://blog.golang.org/context) in most event management procedures, as the telemetry context is passed inside. Seems like a good practice too.
Also enhanced my load test app, it launches several rounds as drawer/player depending on their turn. Need to enhance this a bit, but I can already get some latency results. Those are not good, as all events are launched sequentially in my drawing replay ... There may be other issues as well, need to check this.

[Bios-Marcel]

I am not entirely sure what I am seeing there ^^ It's just the time taken of all events of one type collectively?
Am I missunderstanding? If not, what is this telling you?

[guillaumerosinosky]

Sorry for the delay, I was focusing on other aspects. Each point is an event. Seems that they arrive so fast that the Lobby service does not have time to manage them, especially in the persistence case. I will have to optimize the persistence as we said, and maybe make the load tester more realistic (events are launched one after the other, without any delay; people are not usually this fast 😄).

FYI, after some meetings in my team, we have chosen to enrich the app with scalability on the lobby µservice as well (only having persistence is not enough), in order to have a more realistic µservice application. We'll use a messenging system for that (Kafka to be precise), to permit synchronization between replicas of the Lobby µservice (drawer produce an event message, players will get them). The persistence will still be ensured, by the producer of the message, inside the processing event method of the Lobby. This means that you can have many replicas of the lobby, with a player having WebSocket on whatever replica.