[Enhancement] Add possible monitoring and metrics to decide the lifecycle of the Agent Nodes

[peterzhuamazon]

As a followup to this issue: #494.

Remove an agent node after X runs is a quick fix to remove agent with issues, but it is not fine-grain or robust enough when we have a lot of agents.

We can go ahead and use some custom solutions, such as jenkins plugins, groovy scripts, or even the metrics cluster to find out the current status of the agent nodes. Define a baseline for the health of the agents, and remove them once there is an outage on such node.

Example plugin: https://plugins.jenkins.io/monitoring/
cc: @getsaurabh02 @prudhvigodithi @gaiksaya

Thanks.