cove-oc4ids Ubuntu upgrade

[ghost]

This PR adds code to upgrade cove-oc4ids to Ubuntu 22.04. #381.

The pkgrepo.managed bug is still present on 22.04, for this reason, I have re-configured Salt to install Apache directly from Ubuntu.
IIRC we were only using Ondrej for mod_md support, this functionality is now provided in the native Ubuntu version.

I have also taken this opportunity to implement systemd-networkd network configuration which has proven more resilient on Linode compared to netplan.

Todo:

• Test uWSGI application
• Documentation
• Plan go-live process

[ghost]

@jpmckinney, the cove application is failing to start on this new server (ocp17), please can you investigate?

I have confirmed that the pip packages are all installed correctly.
The uWSGI configuration looks correct too.
There are no errors logged in /var/log/uwsgi/app/cove.log.
It seems to fails to find / start the application.

I was wondering if app changes were needed for the new Python version (3.10) such as pip package updates?

If needed I can downgrade the server to Ubuntu 20.04 to see if that helps.

[jpmckinney]

@jpmckinney, the cove application is failing to start on this new server (ocp17), please can you investigate?

I was wondering if app changes were needed for the new Python version (3.10) such as pip package updates?

Yes, requirements.txt had to be rebuilt. I've done that now.

[jpmckinney]

@jpmckinney, I remember us discussing removing the proxy from the cove servers - which I think will be a good upgrade, simplifying hosting and making server upgrades easier.

What hostnames would you like to use for the cove applications?

I'm thinking review.standard.open-contracting.org and oc4ids-review.standard.open-contracting.org

@yolile What do you think?

[yolile]

I'm thinking review.standard.open-contracting.org and oc4ids-review.standard.open-contracting.org

Maybe review-oc4ids.standard.open-contracting.org instead? So it is read something like "review the oc4ids standard"

[jpmckinney]

review-oc4ids.standard.open-contracting.org is also good with me. (Though technical it reads as review Open Contracting for Infrastructure Data Standards standard).

[ghost]

You can access the pre-live cove application by pointing your hosts file directly to the new server:

176.58.112.127 review-oc4ids.standard.open-contracting.org

I tested the app is running OK and uploaded a file to check functionality but it may be worth someone who knows the app better to check it is functioning all as expected.

Regarding the go-live process, we will need to update anything referencing the old URL to point directly to the new one.
We should also configure a redirect on https://standard.open-contracting.org/infrastructure/review/.
@jpmckinney, please can I leave updating the old URL with you?

[jpmckinney]

You can access the pre-live cove application by pointing your hosts file directly to the new server:

176.58.112.127 review-oc4ids.standard.open-contracting.org

I tested the app is running OK and uploaded a file to check functionality but it may be worth someone who knows the app better to check it is functioning all as expected.

@yolile Do you have the time to test? If not, it sounds like it's working, and we can debug afterwards if something comes up.

Regarding the go-live process, we will need to update anything referencing the old URL to point directly to the new one. We should also configure a redirect on https://standard.open-contracting.org/infrastructure/review/. @jpmckinney, please can I leave updating the old URL with you?

I think I've now made the necessary change in this PR at commit 55423a8

[jpmckinney]

@RobHooper I've made a commit to cove-oc4ids to stop rewriting its own URL as /infrastructure/review.

[yolile]

You can access the pre-live cove application by pointing your hosts file directly to the new server:

It returns the Apache default site for me (even after editing the hosts file)

[jpmckinney]

All looks good - not sure if #409 is a blocker or just something we need to do later.

[ghost]

I'm happy issue #409 is not a blocker.

Go Live Plan:

• Create new site DNS record (oc4ids)
• Create new site DNS record (ocds)
• Merge this PR into Main
• Copy app uploads to new server
• Update app for new URL
• Update sites referring to old URL
• Deploy docs server
• Deploy prometheus server
• Decommission ocp01 and ocp02 on Bytemark

@jpmckinney Rough go-live plan detailed above. Please can you advise on any missing steps.

I believe the cove-oc4ids repo is ready and deployed to the new server. Are changes needed elsewhere?
For instance, I would like the old URL on OCP07 to redirect to the new URL - https://standard.open-contracting.org/infrastructure/review/

[jpmckinney]

I believe the cove-oc4ids repo is ready and deployed to the new server. Are changes needed elsewhere?
For instance, I would like the old URL on OCP07 to redirect to the new URL - https://standard.open-contracting.org/infrastructure/review/

We just need to deploy the ocds-docs server. The necessary changes are in this PR.

Since the redirect will be set up, we don't need to update the old URL(s) immediately. Edit: I've already updated them across our code repositories.

I'll create the DNS records.

[jpmckinney]

review-oc4ids.standard.open-contracting.org is now CNAME to ocp17.open-contracting.org

[ghost]

We just need to deploy the ocds-docs server. The necessary changes are in this PR.

@jpmckinney , I've updated this PR to include the cove3 replacement server too.

You can test the new instance by setting the following in your hosts file:

176.58.107.239 review.standard.open-contracting.org

I've uploaded a test file and it appears to be working OK so I am happy putting new DNS live, if you are?

[jpmckinney]

I've added the CNAME review.standard.open-contracting.org to ocp18, since it doesn't interfere with anything and makes testing easier.

[jpmckinney]

I'm seeing an SSL warning for https://review.standard.open-contracting.org. It's also redirecting to /review/ because I haven't merged open-contracting/cove-ocds#177. I'll merge and re-deploy.

[ghost]

I'm seeing an SSL warning for https://review.standard.open-contracting.org/.

I've just fixed the SSL warning, I had a temporary solution in place until DNS went live.

[jpmckinney]

Hmm, well I merged the PR but now I'm getting jinja2.exceptions.TemplateNotFound: lib.sls errors when trying to deploy anything. (Not related to the changes in this PR – Salt is just broken software.)

[jpmckinney]

Copy app uploads to new server

Why is this crossed out? Doesn't it still need to be done after ocds-docs is deployed?

[jpmckinney]

Update app for new URL
Update sites referring to old URL

I'm not sure what these refer to.

[jpmckinney]

Hmm, well I merged the PR but now I'm getting jinja2.exceptions.TemplateNotFound: lib.sls errors when trying to deploy anything. (Not related to the changes in this PR – Salt is just broken software.)

Ok, I got Salt to work by manually applying the patch saltstack/salt#63184 (needed until salt-ssh==3006).

I deployed cove-ocds, cove-oc4ids, docs and prometheus.

Copy app uploads to new server

Why is this crossed out? Doesn't it still need to be done after ocds-docs is deployed?

I've copied the media directory and the db.sqlite3 file to cove-ocds, as documented.

There were only 5 reports in the old cove-oc4ids, so I didn't bother copying them over to the new server.

[jpmckinney]

I've added decommissioning the old servers as a step to the go-live plan.

[ghost]

Awesome this is coming together well 👍

Well done finding the bug in salt. I haven't been affected by this yet, probably quirks from different software versions on my system.

Update app for new URL

This is to update the cove application for the new URL.

You've actioned this, I believe these are the corresponding commits:
open-contracting/cove-ocds#177
open-contracting/cove-oc4ids@315adaa

Update sites referring to old URL

This is to update anything that points to the cove review applications.

For instance the "Data Review Tool" button on the standard site: https://standard.open-contracting.org/latest/en/
I am not sure how many links are around still configured for the old URL.

Since we have redirects configured, this isn't a problem but it would be good to fix them as we find them.
Extra, unnecessary, redirects will slow down site visitors - adding an extra request whenever they visit the page.

Copy app uploads to new server

Thanks for sorting app media, I had the impression (incorrectly) that this step wasn't needed but it's all sorted now.

Decommission ocp01 and ocp02 on Bytemark

I will action our standard decommissioning process.
Shutting the instances down for two weeks and then terminating.

Would you like a final backup of anything on these servers? I assume everything we'd want is already in Github etc.

[jpmckinney]

This is to update anything that points to the cove review applications.

Aha, I've done that for our GitHub repositories - the updates to the standard documentation are in the next release (TBD). Otherwise, there are Google Docs, etc. that refer to the old URL, but we can change them as we find them.

Would you like a final backup of anything on these servers?

I think everything is on GitHub or transferred to the new server :) (/media, db.sqlite3)