Anyone doing carbon aware scheduling with Kubernetes?

[rossf7]

Hi,
I'm interested in making software carbon aware. There are many ways of doing this but Kubernetes is widely used, provides APIs for scheduling and can be extended using CRDs (Custom Resource Definitions) and custom controllers.

For temporal shifting (moving workloads to times when carbon intensity is lower) there is the carbon-aware-keda-operator developed by Microsoft which builds on top of the KEDA (Kubernetes Event Driven Autoscaling) project. It can do autoscaling based on many types of metrics such as SQS or Kafka queue length.

I've been looking at spatial shifting (moving workloads to physical locations where carbon intensity is lower). I've created a proof of concept carbon-aware-karmada-operator that extends the Karmada project which can do multi-cluster and multi-cloud scheduling.

Are there other projects out there doing this? Are you already doing carbon aware scheduling in your clusters?

I'm also interested to hear what this community thinks to the current efforts? As well as suggestions on how they can be improved and how we can increase adoption. Thanks!

[AydinMirMohammadi]

We plan an operator for time shifting. I think the keda operator is more suitable for demand shaping (do less and do more).
I'm also interested in opinions about location shifting. Because we have also pay attention for the embodied carbon. So optimizing scope 2 against scope 3.
Maybe we need a concept which will optimize scope 2 and scope 3. Free resources within a grid with minimize carbon

[rossf7]

Hi @AydinMirMohammadi, that's great to hear you're also looking at this area.

Good point the carbon-aware-keda-operator is more doing demand shaping by setting the max replicas depending on the carbon intensity. This is because the number of replicas is still controlled by the keda scaler being used.

For a time shifting operator have you seen the exporter the keda operator uses? It fetches the carbon intensity forecast using the carbon-aware-sdk from the GSF and saves it to a configmap that can be read by other components.

For location shifting you're right the karmada solution I've been working on just considers scope 2 emissions from the electricity grid. The scope 3 emissions especially the embodied carbon in the servers need to be considered but it's hard to get data for this.

I hope the real time carbon metrics project will help with this https://github.com/orgs/Green-Software-Foundation/discussions/34

[AydinMirMohammadi]

Jep. We need the metrics from the underlying providers. For Europ, I provide open data for the Carbon Aware SDK. The data is from Energy Charts a Frauenhofer project. See https://carbon-aware-computing.com

And yes - the configmap is the right place for that.

Let's stay in contact. I will report progress here

[mrchrisadams]

hi @AydinMirMohammadi - how current is the data you generate, and does it include figures for consumption of electricity? That last one has typically been hard to find in my experience.

[AydinMirMohammadi]

hi @mrchrisadams - the data is provided by Energy Charts a service of the Frauenhofer ISE, which is the data of entos-e and converted by me into the SDK-Format.

The timeseries are created hourly by Energy Charts but I grab them about five times a day. The forecast does not change, but the current. The forecast is generated at 19:00 Brussel time. The data has not the same accuracy for all Europeans countries, this is the reason they want to add some additional data sources and long term forecasts based on weather data. Q4/23 or Q1/24

The data I use is the amount of renewable energy in production not load (not consumption). It is not the carbon intensity.

The guys are very open and are highly interested to provide us with data for Green Software Development and I'm in contact with them. This is the first MVP and I don't want unnecessary data processing and additional interfaces for me.

If you want, we could have a short call an figure out extra need. For me - the next data interface is the current grid carbon intensity for my monitoring project. A open data replacement of co2signal. The calculation of the grid carbon intensity is easy based on that data (but you need the raw data).

[mrchrisadams]

Hi @AydinMirMohammadi - is the work you've done so far in the public domain anywhere?

I wasn't aware that German data was published under an open license in close to real time resolution, and the combination of different data sources, each with different licensing terms make me wonder if can be licensed as open data.

We ended up looking into this and diving into sui generis rights in some other related work last year. If there are sufficient changes being made in the processing/ETL stages, my understanding is that you can make an argument that yes, the new data can be relicensed as open data, even if some of the sources are not themselves open data in the first place.

I'm basing my question on this text here from the Energy Chart site, which largely says that the data isn't just from ENTSO-E (open data) but also other providers where the licensing is less clear:

Stromampel für Deutschland

Datenquelle: ENTSO-E, AGEE-Stat, Destatis, Fraunhofer ISE, AG Energiebilanzen, EEX

Letztes Update: 01/10/2024, 9:39 AM GMT+1

[AydinMirMohammadi]

I have an informal agreement with the Fraunhofer ISE for publishing the data they provided. This is licensed as CC0. I also provide the data for UK and this is published under CC BY 4.0 DEED.

[JacobValdemar]

Hey, sorry I am late to the party 😅 I just wanted to add that I am starting my Masters Thesis project this Monday (28 Aug '23), running to end of year, with the aim of creating a version of Karpenter that can perform carbon aware autoscaling of Kubernetes clusters. I don't really have much to show yet, and the first weeks will probably just be gathering knowledge. But I am very happy to see that you folks also find this topic interesting. I am excited to get started, and the work by you, and other people behind the Green Software Foundation has already been, and continues to be, tremendously helpful. 💚

[rossf7]

Hey @JacobValdemar, that's fantastic to hear you're doing your masters thesis on this topic. Karpenter is a great tool with a lot of potential for making it carbon aware. Please keep us updated on how it goes and let us know if we can help somehow.

[JacobValdemar]

Thank you @rossf7 for offering your help! I have read some of your blog posts on the topic and they have been really helpful! All the links you provide enable me to discover new tools and knowledge sources.

[rossf7]

An update on the carbon-aware-karmada-operator. I demoed the prototype I developed at the Karmada community meeting a couple of weeks ago. https://www.youtube.com/watch?v=-Y0SDbSKHJk

I got some great feedback including that it would be useful to support weights when scheduling across clusters.
https://karmada.io/docs/userguide/scheduling/resource-propagating/#multiple-strategies-of-replica-scheduling

This issue was created as a result karmada-io/karmada#3917

I think a good next step would be to see if there are members of the Karmada community and / or the community here that would like to collaborate on this. Please do chime in if you're interested 🙏 and I'll keep a 🧵 here updated.