Upgrade servers to Ubuntu 20.04

[jpmckinney]

Once available on Bytemark. Document the process.

Can consider moving someMove servers to Linode, if upgrading involves provisioning a new server. (Bytemark has very responsive support; I haven't had to interact with Linode recently.)

Then, change the Ubuntu version in GitHub Actions workflows:

• cove-oc4ids (to 22.04)
• cove-ocds (to 22.04)
• kingfisher-collect
• kingfisher-process
• kingfisher-summarize

[jpmckinney]

Upgrade servers to Ubuntu 20.04

[ghost]

Ubuntu 20.04.01 (the point release that marks this as production ready) should be released on 06 Aug 2020. I imagine Bytemark will start supporting it from then.
https://wiki.ubuntu.com/FocalFossa/ReleaseSchedule

[jpmckinney]

Noting some changes we'll likely have to make:

OpenDataServices/opendataservices-deploy@ebf4949

[Bjwebb]

BTW, there's still no option to start a server with Ubuntu 20.04 on Bytemark. We've been testing it out by creating a 18.04 server and upgrading it.

[ghost]

I reached out to Bytemark support and they have no plans to add Ubuntu 20.04 to their list. ☹️
They also are not planning on removing the Ubuntu 14.04 which has been end of life for months.

We should be able to workaround this using their custom ISO option.

[jpmckinney]

@RobHooper Are there pros/cons to (incrementally) moving to a similar host that does offer Ubuntu 20.04?

We use a Linode VPS for the CRM, and we have a Hetzner account for dedicated servers, which we can also use for their Cloud offering. I don't know how the hardware/performance and support compare.

[ghost]

It's definitely worth looking at other providers and seeing how they compare.
We've been very happy with Linode - there are lots of hosting options out there so Ubuntu 20.04 sounds like a good time to review our options.

Its worth considering the flow of data between servers, especially when moving incrementally because bytemark to bytemark networking is will be very quick, the data stays in the same data centre probably moving at gigabit speeds. Bytemark to Linode (or anywhere else) will be slower, possibly only milliseconds slower but if you're sending a lot of data there is a bigger impact.

[jpmckinney]

All our Bytemark servers use less than 20GB, so not too much data.

Besides Prometheus clients, the flows are:

• PostgreSQL replication between kingfisher-process and kingfisher-replica
• PostgreSQL reads between redash and kingfisher-replica
• Proxy requests from docs to cove-ocds and cove-oc4ids

standard-search currently builds an Elasticsearch index by crawling docs over the web, but this is inefficient, and I'm working on it building the index locally (during docs deployment), so that there won't be any data flow between them.

[jpmckinney]

Looking ahead, we're expecting a new data registry web service to read from kingfisher-replica, and to make requests to the services running on kingfisher-process (to initiate data collection and data processing). I haven't looked into whether Hetzner Cloud servers are in the same datacenters as dedicated servers, but if so this might be a reason to consider Hetzner Cloud. We don't have server specs for the registry, but right now I'm thinking it doesn't need a dedicated server.

We're also expecting a flattening tool web service to read from kingfisher-replica.

For the other services like Toucan, etc. which can be hosted anywhere, I still have a preference for fewer hosting providers, so ideally we would keep it to 3 like we have now (Hetzner, Bytemark, Linode).

[jpmckinney]

Noting that we might want to address #35 when moving to new servers.

[Bjwebb]

@RobHooper Are there pros/cons to (incrementally) moving to a similar host that does offer Ubuntu 20.04?

I'd agree this is worth looking into, and it is something Open Data Services may also be looking at for our own servers, since some of the reasons we chose Bytemark originally may no longer be true (particularly since they were aquired).

[ghost]

If anyone does want to create an Ubuntu 20.04 server on Bytemark, you can do so by using the following custom ISO at the operating system step of server provisioning.
https://releases.ubuntu.com/20.04/ubuntu-20.04.1-live-server-amd64.iso

You'll then need to open up the console and follow through the Ubuntu install steps.
Just a heads up, it was quite a slow install, that said I was using a tiny server which didn't help.

[jpmckinney]

Regarding choice of host, I think the differences between the good providers are fairly minimal, and not important enough. I've seen good reviews for UpCloud and Vultr in terms of performance, but Dogsbody hasn't used them yet, so I prefer a host we know. DigitalOcean is also good, but we don't have an account with them. Dogsbody also mentioned Exoscale. All of these have official support on https://registry.terraform.io with respect to #156.

Since we already have an account with Linode, I'm fine to continue with them. We use it for the CRM, and The Idea Bureau uses it for www.open-contracting.org.

If we return to this question, I had started this spreadsheet in 2019.

[ghost]

I've just raised a PR improving the Ubuntu 20.04 core build #281
I'd like to update the documentation on where to create new Ubuntu 20.04 instances, just double checking we are proceeding with Linode?

[jpmckinney]

Yes, let's go ahead with Linode.

[jpmckinney]

@RobHooper Per the email thread with Dan:

Linode are starting to creak in our humble opinion. There have been a number of networking issues that we have been working with Linode on over the last few months. The outages over the last few days haven't been the only "oops" moments we've had from Linode's side.

This maybe due to their purchase by Akamai in May this year. These outages may also be the exact kick they need to pull their act together.

There is absolutely no reason to throw baby out with the bathwater. If Linode works for you then we are very happy to continue using them and supporting you with them. We are just using this as a reminder that other vendors exist and we should make sure we continue to point our customers to the best vendors for their needs.

Certainly, compared Bytemark we are very happy that the move from Bytemark to Linode will be an improvement. We are also currently evaluating Digital Ocean and Vultr as closely equivalent providers.

We only have two servers at Linode and haven't moved the others yet. If in your opinion Digital Ocean is a better choice, let's move the servers there instead.

[ghost]

@jpmckinney We've had a discussion internally and we are happy continuing with Linode for now.
There is no way of knowing which ISP will be better in the long run.

Dan's message above was mainly a heads up in-case there were more issues but we are not at the point of recommending customers move away from Linode yet.

That said, this could be a good prompt to test your stack on Digital Ocean making it easier to deploy there in the future. We could also investigate splitting hosting between the two ISPs.
This is a new, separate task if you wanted to proceed in this direction.

[jpmckinney]

@RobHooper Sounds good, let's create an issue for that, but for now let's continue to upgrade Ubuntu and move to Linode.

[jpmckinney]

Closing in favor of #381, since we've been upgrading to 22.04.