improve provider observability

[troian]

Is your feature request related to a problem? Please describe.

the issue

Inventory operator

Extend inventory operator to provide not only inventory availability, but health of all key components:

• egress

  • monitor network speeds and latency @chainzero

• hardware

  • GPUs (device driver plugins)
  • storage devices

• cluster

  • cluster status
  • node(s) status

Ceph

• does Rook/Ceph have API to check state of the ceph cluster @chainzero
• limit amount of mounted persistent volumes (at moment it is set to unlimited)
• limit total amount of PV on single kube cluster
• test IOPS
  • test for max IOPS at provider provisioning stage
  • periodically test for current IOPS
• check if there is a way to limit IOPS per workload

[troian]

there is a CEPH API
have to check how details information can be obtained, does it support streaming (websockets)

[chainzero]

Along with the CEPH REST API referenced by @troian above there is:

rook-ceph includes Prometheus Metrics in default install.

Example raw output from an OCL managed provider:
https://gist.github.com/chainzero/be7a39f2a9ab8d663420f8c8441a2090

Summary of analytic categories:
https://gist.github.com/chainzero/762dc0aa3b07e03b2f0ec3faf29a09a0