diff --git a/content/blog/2024/09/10/2024-09-10-update-center-brownouts-3.adoc b/content/blog/2024/09/10/2024-09-10-update-center-brownouts-3.adoc new file mode 100644 index 000000000000..f4f1697b47f1 --- /dev/null +++ b/content/blog/2024/09/10/2024-09-10-update-center-brownouts-3.adoc @@ -0,0 +1,62 @@ +--- +layout: post +title: "Brownout on Update Center (updates.jenkins.io):\n 11-12 September 2024" +tags: +- jenkins +- jenkins-infra +- update-center +authors: +- smerle33 +opengraph: + image: /images/post-images/2023/01/12/jenkins-newsletter/infrastructure.png +overview: "We'll be using the new Update Center implementation in production for 1 day" +discourse: true +--- + +== Summary (link:https://en.wikipedia.org/wiki/Wikipedia:Too_long;_didn%27t_read[TL;DR]) + +The service https://updates.jenkins.io will switch its implementation to a new system during 1 day: + +- Wednesday 11 September 2024 from 02:00pm UTC until Thursday 12 September 2024 02:00pm UTC + +All Jenkins users are impacted but should not see any functional change. + +⚠️ Please, check that your organization respects the advertised DNS TTL or you might be stuck in the brownout longer that expected. + +Under the hood, any HTTP request made to this service will be redirected to a mirror close to user locations during these brownouts. + +== What is the "Update Center"? + +Jenkins link:https://updates.jenkins.io[Update Center] is a web server at the core of the Jenkins public infrastructure which distributes the plugins, tool installers, and versions index to all Jenkins servers all across the world. + +From the Wizard installation to regular plugin updates, if you run Jenkins then you use this service under the hood. + +Today, it serves around 50 Tb of data (outbound bandwidth) each month from a single Virtual Machine on AWS, which costs around $6,000 per month. + +In order to sustain this service and improve it, the Jenkins infrastructure team has worked relentlessly during the past years to have a new sustainable implementation for this service. + +The new Update Center implementation features a highly available system, which redirects user requests to a download mirror close to their location. +There is lots of details and additional information that you can read further about in the following link:https://github.com/jenkins-infra/helpdesk/issues/2649[GitHub issue]. + +== Why this "link:https://en.wikipedia.org/wiki/Brownout_(electricity)[Brownout]"? + +We have reached the point where our functional/performance tests are meeting our expectation: + +- All of our Jenkins controllers are using the new Update Center implementation. +- We are using the new updates center (with DNS override) in our own networks in different geo-locations. +- All of our Jenkins Docker image are using the new system. +- Our initial performance tests +- After 2 successful 1-hour brownouts + +== How + +Both current and new Update Centers are updated at the same time and serve the same index. + +Starting 5 weeks ago, the Jenkins infrastructure has been using the new Update Center (https://azure.updates.jenkins.io) with a client-side DNS override (`updates.jenkins.io` hostname points to this new service). + +During this brownout, we'll simply switch the DNS entry `updates.jenkins.io` to this new service and watch for the logs and error rate. +At the end, we'll switch DNS back to the normal service and then analyze metrics and logs to see how the system behaved. + +We are confident the new system will perform as expected. We now want to execute a 1 day brownout. + +Please refer to the https://github.com/jenkins-infra/helpdesk/issues/2649[helpdesk ticket] for more information.