Add 65k GKE benchmark

[besher-massri]

This PR adds a benchmark for GKE on 65,000 nodes scale

This is following up on the recent GKE announcement for support clusters with 65,000-nodes scale.

It benchmarks GKE on a (simulated) AI workload using using Terraform and ClusterLoader2.

I didn't include it as part of benchmarks directory as the benchmarking process here is different (using CL2) and has a different infra setup.

[achandrasekar]

Noticed you mentioned the infra setup and cluster loader tool makes this different to the other benchmarks in the benchmarks directory. It might still make sense to consolidate under the benchmarks directory where infra setup could go under infra - maybe even a separate 65k folder and cl2 and workload automation could go under the benchmark/tools directory.

[andrewsykim]

Agree with @achandrasekar that it would be good to put this under the existing benchmarks/ directory. Is it possible to make that change?

I didn't include it as part of benchmarks directory as the benchmarking process here is different (using CL2) and has a different infra setup.

I think tooling being different is not a blocker, but would be good to consoildiate eventually

[besher-massri]

Agree with Ashok Chandrasekar that it would be good to put this under the existing benchmarks/ directory. Is it possible to make that change?

I didn't include it as part of benchmarks directory as the benchmarking process here is different (using CL2) and has a different infra setup.

I think tooling being different is not a blocker, but would be good to consoildiate eventually

I had an attempt at reorganizing the benchmark, here is a summary of the changes:

• put existing infra under infra/gpu-cluster
• moved infra part under infra, put under folder 65k-cpu-cluster
• moved CL2 part under benchmark/tools
• renamed current README to gpu-benchmark.md
• moved 65k README and renamed into 65k-nodes-simulated-ai-workload
• added a root level README for benchmarks/ directory

[achandrasekar]

Thanks for refactoring @besher-massri! @annapendleton can you take a look to make sure this doesn't affect the existing benchmarks?

[annapendleton]

This is awesome work! Everything lgtm and shouldn't break existing functionality since it's just migrating locations. Only thing is we want to accurately represent what the infra does so people understand that it supports both GPU and TPU infra

[besher-massri]

Thanks for the review and comments! I pushed another commit, changing from gpu to accelerator and highlighting that both GPU and TPU clusters are supported. Changed some naming, minor rewording as well. Let me know if it's okay.

[annapendleton]

That looks great, thanks! Please make sure to resolve anything lingering with Andrew prior to submission :)

[andrewsykim]

LGTM, deferring merge to @genlu2011

[andrewsykim]

/gcbrun