feat: added dependency support to k8s #1487
Conversation
AvineshTripathi
requested review from
mlabouardy,
greghub and
Azanul
as code owners
|
We can create a routine which will start when all the resources(k8s) from the wait group are completed |
AvineshTripathi
force-pushed
the
k8s-dep-graph
branch
from
bad2bf4
to
4ce20fb
Compare
| @@ -37,7 +37,7 @@ func FetchResources(ctx context.Context, client providers.ProviderClient, db *bu | |||
| log.Printf("[%s][K8s] %s", client.Name, err) | |||
| } else { | |||
| for _, resource := range resources { | |||
| _, err = db.NewInsert().Model(&resource).On("CONFLICT (resource_id) DO UPDATE").Set("cost = EXCLUDED.cost").Exec(context.Background()) | |||
| _, err = db.NewInsert().Model(&resource).On("CONFLICT (resource_id) DO UPDATE").Set("cost = EXCLUDED.cost, relations=EXCLUDED.relations").Exec(context.Background()) | |||
There was a problem hiding this comment.
if the resource or the relation was deleted, in the next cycle it will not be updated if this is not added
There was a problem hiding this comment.
cost = EXCLUDED.cost means that if a conflict occurs, the cost column of the existing row will be updated with the value from the new row being inserted. Same goes for relations
There was a problem hiding this comment.
thats correct and we want to ensure this field gets updated
Since we have to get list of all the services to find the relation between pod and services, a new call to the service list has to be done. However we already have a service fetcher and since services might be used for other relations, i felt it would be better if we cache it for some time to avoid multiple calls. This I feel can be used across the project to find relations where presence of other resource is mandatory.
Will do
where should we document |
Is the case different for nodes, pods, etc.?
Let's create a documentation page for dependency graph that'll help users to navigate the graph easily. Like a user guide |
Didnt get you here |
What's stopping us from caching nodes and pods too? |
|
In case of nodes and pod we do get fields to get the relation in there as comparison is not required while in case of service its not the case. Let me know if that answers your question |
It does, thanks |
AvineshTripathi
force-pushed
the
k8s-dep-graph
branch
from
4ce20fb
to
0e1a2cf
Compare
Because we currently fetch all the objects separately and without any connection, some relationships are difficult to generate and would result in redundancy. In other cases, we also need both resources to be present in order to validate the relations.
To avoid this, we should have something like a stitching module that would asynchronously join the node based on a predefined condition. Open to discussion
Solution proposed:
Based on current architecture, the relationship field is sent by the resource itself. However, sometimes comparison between two resources are required to find a relationship. The solution we are using for this PR (which may change if any better approach to this problem comes) is that if a resource requires another resource to find the relationship, it will do so by making an api call and fetching all the resources of that particular service where the relationship is to be found. However, the result is cached and stored in memory for some time, so if another resource requires this resource list can use it.