Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Noisy neighbours causing CPU entitlement usage of consistent load apps to increase #345

Open
ctlong opened this issue Feb 28, 2024 · 0 comments

Comments

@ctlong
Copy link
Member

ctlong commented Feb 28, 2024

Description

We were running an app using a consistent amount of CPU, which was under its CPU entitlement, and expected that the addition of a noisy neighbour (an app that uses a lot of CPU) to the system would not affect the original app. However, instead we saw that the original app increased the amount of CPU it was using.

Environment

  • garden-runc-release version: ?
  • Enabled configuration: ?
  • IaaS: GCP
  • Stemcell version: ?
  • Kernel version: ?

Steps to reproduce

Here is the app we deployed that used a consistent amount of CPU:

package main

import (
        "crypto/rand"
        "crypto/rsa"
        "fmt"
        "net/http"
        "os"
)

func main() {
        http.HandleFunc("/", doWork)
        fmt.Println("listening...")
        err := http.ListenAndServe(":"+os.Getenv("PORT"), nil)
        if err != nil {
                panic(err)
        }
}

func doWork(res http.ResponseWriter, req *http.Request) {
        _, err := rsa.GenerateKey(rand.Reader, 1024)
        if err != nil {
                http.Error(res, err.Error(), http.StatusInternalServerError)
        }
}

We hit that app with a load-generating tool, e.g. hey. The command we used was: hey -q 2 -z 60m https://cpu-load-generator.example.com/. Scale the app as necessary to ensure that it remains under its entitlement.

Then we deployed dora and hit its stress_testers endpoint to create a noisy neighbour.

Once that's done you should see dora crushing the cores on your Diego Cell(/s). In addition, you should be able to observe that the CPU use of the original app increases.

Logs

None

Docker image (if relevant)

N/A

Cause

Not sure

Resolution

Not sure

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Pending Review | Discussion
Development

No branches or pull requests

1 participant