Optimize tf template E2E deployment time

[yiyinglovecoding]

Making terraform apply faster by paralleling creation of gke cluster and cloudsql database

• Making custom-network a separate module and module infra should depends on custom-network in rag
• Making cloudsql-secret a separate resource that depends on namespace and cloudsql

It doesn't affect Jupyterhub and Ray using Infra module as the infra still create network for them. In RAG, however, custom-network willl create network outside infra module.

test:

module.infra[0].module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-nodeService_account[0]: Creation complete after 9s [id=yiyingzhang-gke-dev/roles/container.nodeServiceAccount/serviceAccount:tf-gke-test-refactor12-6wm8@yiyingzhang-gke-dev.iam.gserviceaccount.com]
module.cloudsql.module.cloudsql.google_sql_database_instance.default: Still creating... [10s elapsed]
module.infra[0].module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still creating... [10s elapsed]
module.cloudsql.module.cloudsql.google_sql_database_instance.default: Still creating... [20s elapsed]
module.infra[0].module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still creating... [20s elapsed]
module.cloudsql.module.cloudsql.google_sql_database_instance.default: Still creating... [30s elapsed]
module.infra[0].module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still creating... [30s elapsed]
module.cloudsql.module.cloudsql.google_sql_database_instance.default: Still creating... [40s elapsed]
module.infra[0].module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still creating... [40s elapsed]
module.cloudsql.module.cloudsql.google_sql_database_instance.default: Still creating... [50s elapsed]
module.infra[0].module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still creating... [50s elapsed]
module.cloudsql.module.cloudsql.google_sql_database_instance.default: Still creating... [1m0s elapsed]
module.infra[0].module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still creating... [1m0s elapsed]
module.cloudsql.module.cloudsql.google_sql_database_instance.default: Still creating... [1m10s elapsed]
module.infra[0].module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still creating... [1m10s elapsed]
module.cloudsql.module.cloudsql.google_sql_database_instance.default: Still creating... [1m20s elapsed]
module.infra[0].module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still creating... [1m20s elapsed]
module.cloudsql.module.cloudsql.google_sql_database_instance.default: Still creating... [1m30s elapsed]
module.infra[0].module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still creating... [1m30s elapsed]
module.cloudsql.module.cloudsql.google_sql_database_instance.default: Still creating... [1m40s elapsed]
module.infra[0].module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still creating... [1m40s elapsed]
module.cloudsql.module.cloudsql.google_sql_database_instance.default: Still creating... [1m50s elapsed]
module.infra[0].module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still creating... [1m50s elapsed]
module.cloudsql.module.cloudsql.google_sql_database_instance.default: Still creating... [2m0s elapsed]
module.infra[0].module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still creating... [2m0s elapsed]
module.cloudsql.module.cloudsql.google_sql_database_instance.default: Still creating... [2m10s elapsed]
module.infra[0].module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still creating... [2m10s elapsed]
module.cloudsql.module.cloudsql.google_sql_database_instance.default: Still creating... [2m20s elapsed]
module.infra[0].module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still creating... [2m20s elapsed]
module.cloudsql.module.cloudsql.google_sql_database_instance.default: Still creating... [2m30s elapsed]
module.infra[0].module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still cr

time terrform apply on AP cluster
before: around 30min
after: around 20min

e2e: done

[yiyinglovecoding]

/gcbrun

[yiyinglovecoding]

/gcbrun

[yiyinglovecoding]

test ray cluster failed with gmp engine error, seems not related to my change.
https://screenshot.googleplex.com/BFBmgVbDrgAHbNi

[yiyinglovecoding]

/gcbrun

[yiyinglovecoding]

/gcbrun

[yiyinglovecoding]

/gcbrun

[yiyinglovecoding]

/gcbrun

[yiyinglovecoding]

/gcbrun

[yiyinglovecoding]

/gcbrun

[andrewsykim]

I think we want to test network creation as part of infrastrcuture/ since that's how it'll be used for RAG, Ray, Jupyter. Does it have to be a separate step in the tests?

[yiyinglovecoding]

yes you are right, combined it into cluster creation

[yiyinglovecoding]

a I might misunderstood it so want to confirm that
if we want to accelerate the deployment by paralleling cloudsql database and gke cluster creation, we have to move network creation out of infrastrcuture/.
As cloudsql database depends on the network.
@imreddy13

[imreddy13]

@andrewsykim this PR splits out the network creation from infra, because for RAG we need to create the cloud SQL instance in parallel to the GKE cluster (which needs a network to exist).

But for ray and jupyter, we don't need that. @yiyinglovecoding can we test both flows in our E2E?

1. network created by infra for ray and jupyter applications
2. network created by rag for rag application

Might need to restructure and/or add tests

[andrewsykim]

I think that could add a lot of complexity. How much faster is creation due to this change? If it's not significant should we reconsider?

[yiyinglovecoding]

it will be around at least 10mins for AP.

There has been lots of changes in the template so I just ran the
the AP creation before change is around 25mins, after change it would be 20mins.

@imreddy13

[yiyinglovecoding]

/gcbrun

[yiyinglovecoding]

/gcbrun

[yiyinglovecoding]

/gcbrun

[imreddy13]

@andrewsykim this PR splits out the network creation from infra, because for RAG we need to create the cloud SQL instance in parallel to the GKE cluster (which needs a network to exist).

But for ray and jupyter, we don't need that. @yiyinglovecoding can we test both flows in our E2E?

1. network created by infra for ray and jupyter applications
2. network created by rag for rag application

Might need to restructure and/or add tests

[yiyinglovecoding]

/gcbrun

[yiyinglovecoding]

/gcbrun

[yiyinglovecoding]

/gcbrun

[spencer-p]

Closing old issues with merge conflicts. Please rebase and re-open if still relevant.