Allow to set up load balancing when installing DOMjudge.

[nickygerritsen]

When DOMjudge/domjudge#1654 is merged, we store sessions in the database and thus we can have a real loadbalanced set up. This PR enables this, put behind a feature flag (since it doesn't make sense if you only have one machine).

This PR does a couple of things:

• It allows one to create a DBA user, which we need since we no longer use localhost to connect to the database in the laodbalancing scenario.
• Makes sure the domjudge DB user can also connect from external IP's in the configured range.
• Adds an extra nginx listen on port 81, which accepts connections from nginx on port 80. The one on port 80 loadbalances to all domservers.

I tested this on the DB sessions branch and it seems to work.

Since we already had this keepalived floating IP for the database, we can use that as the 'frontend IP' teams and the judgehost uses, but we already did that.

nginx itself takes care of not sending traffic to a loadbalanced server if it is down, so we don't have to do anything for that.

I would like to use this at the World Finals in Dhaka, and test it earlier (at EAPC and BAPC if we have > 1 server there or otherwise at the integration test).

[vmcj]

You change the order of some roles so does

domjudge-scripts/provision-contest/ansible/roles/domserver/tasks/main.yml

Line 22 in 9f1a7ab

# When using replication, the DB will be dropped and recreated on the slave later.

still apply (I assume so as we start the script later..)

[nickygerritsen]

You change the order of some roles so does

domjudge-scripts/provision-contest/ansible/roles/domserver/tasks/main.yml

Line 22 in 9f1a7ab

# When using replication, the DB will be dropped and recreated on the slave later.

still apply (I assume so as we start the script later..)

Yes

[eldering]

Suggested change

	proxy_set_header Connection "";
	proxy_set_header Connection "";

[eldering]

A caution note: in the WF setup each server runs its own master-master replicated DB, so if you use load balancing this way, then you write to both masters and replication will happen both ways. I'd make sure to stress test this, as I don't know if there's a guarantee that both servers will end up with the exact same replication log or how MySQL performance is impacted if it does some kind of locking to guarantee this.

I guess the alternative would be to redirect only read-only traffic to the "secondary" server, which would also have to include any login/out actions due to the session management.

[nickygerritsen]

A caution note: in the WF setup each server runs its own master-master replicated DB, so if you use load balancing this way, then you write to both masters and replication will happen both ways.

No, when this feature is enabled I explicitly write to only one of the database servers. Or well, I write to the keepalived IP, so if the IP moves, then indeed it writes to the other one. But it should never write to both 'at the same time'.

I'd make sure to stress test this, as I don't know if there's a guarantee that both servers will end up with the exact same replication log or how MySQL performance is impacted if it does some kind of locking to guarantee this.

I guess the alternative would be to redirect only read-only traffic to the "secondary" server, which would also have to include any login/out actions due to the session management.

Sessions are stored in the databse, so they are not read-only?

[eldering]

A caution note: in the WF setup each server runs its own master-master replicated DB, so if you use load balancing this way, then you write to both masters and replication will happen both ways.

No, when this feature is enabled I explicitly write to only one of the database servers. Or well, I write to the keepalived IP, so if the IP moves, then indeed it writes to the other one. But it should never write to both 'at the same time'.

Ah, ok. Then ignore my remarks.

I'd make sure to stress test this, as I don't know if there's a guarantee that both servers will end up with the exact same replication log or how MySQL performance is impacted if it does some kind of locking to guarantee this.
I guess the alternative would be to redirect only read-only traffic to the "secondary" server, which would also have to include any login/out actions due to the session management.

Sessions are stored in the databse, so they are not read-only?

Never mind this part, it's not relevant anymore given the above. And I made a typo and should have written "exclude any login/out actions".