Skip to content

Commit

Permalink
feat(blog): new blogpost for the uc-brownout-3
Browse files Browse the repository at this point in the history
  • Loading branch information
smerle33 committed Sep 10, 2024
1 parent b3651bb commit 99ec44c
Showing 1 changed file with 62 additions and 0 deletions.
62 changes: 62 additions & 0 deletions content/blog/2024/09/10/2024-09-10-update-center-brownouts-3.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
---
layout: post
title: "Brownout on Update Center (updates.jenkins.io):\n 11-12 September 2024"
tags:
- jenkins
- jenkins-infra
- update-center
authors:
- smerle33
opengraph:
image: /images/post-images/2023/01/12/jenkins-newsletter/infrastructure.png
overview: "We'll be using the new Update Center implementation in production for 1 day"
discourse: true
---

== Summary (link:https://en.wikipedia.org/wiki/Wikipedia:Too_long;_didn%27t_read[TL;DR])

The service https://updates.jenkins.io will switch its implementation to a new system during 1 day:

- Wednesday 11 September 2024 from 02:00pm UTC until Thursday 12 September 2024 02:00pm UTC

All Jenkins users are impacted but should not see any functional change.

⚠️ Please, check that your organization respects the advertised DNS TTL or you might be stuck in the brownout longer that expected.

Under the hood, any HTTP request made to this service will be redirected to a mirror close to user locations during these brownouts.

== What is the "Update Center"?

Jenkins link:https://updates.jenkins.io[Update Center] is a web server at the core of the Jenkins public infrastructure which distributes the plugins, tool installers, and versions index to all Jenkins servers all across the world.

From the Wizard installation to regular plugin updates, if you run Jenkins then you use this service under the hood.

Today, it serves around 50 Tb of data (outbound bandwidth) each month from a single Virtual Machine on AWS, which costs around $6,000 per month.

In order to sustain this service and improve it, the Jenkins infrastructure team has worked relentlessly during the past years to have a new sustainable implementation for this service.

The new Update Center implementation features a highly available system, which redirects user requests to a download mirror close to their location.
There is lots of details and additional information that you can read further about in the following link:https://github.com/jenkins-infra/helpdesk/issues/2649[GitHub issue].

== Why this "link:https://en.wikipedia.org/wiki/Brownout_(electricity)[Brownout]"?

We have reached the point where our functional/performance tests are meeting our expectation:

- All of our Jenkins controllers are using the new Update Center implementation.
- We are using the new updates center (with DNS override) in our own networks in different geo-locations.
- All of our Jenkins Docker image are using the new system.
- Our initial performance tests
- After 2 successful 1-hour brownouts

== How

Both current and new Update Centers are updated at the same time and serve the same index.

Starting 5 weeks ago, the Jenkins infrastructure has been using the new Update Center (https://azure.updates.jenkins.io) with a client-side DNS override (`updates.jenkins.io` hostname points to this new service).

During this brownout, we'll simply switch the DNS entry `updates.jenkins.io` to this new service and watch for the logs and error rate.
At the end, we'll switch DNS back to the normal service and then analyze metrics and logs to see how the system behaved.

We are confident the new system will perform as expected. We now want to execute a 1 day brownout.

Please refer to the https://github.com/jenkins-infra/helpdesk/issues/2649[helpdesk ticket] for more information.

0 comments on commit 99ec44c

Please sign in to comment.