-
Notifications
You must be signed in to change notification settings - Fork 0
Home
The hokiegeek2 Arkouda wiki contains descriptions of containerized Arkouda, Arkouda-on-Kubernetes, and Arkouda-on-Slurm integtrated with Kubernetes environments. In addition, an enterprise version of Arkouda that integrates with Prometheus and Grafana to provide Kubernetes native metrics, alerting, and eventually, telemetry is discussed.
Arkouda-On-Kubernetes describes packaging of the Arkouda client and server in separate Docker containers and subsequent deployment to Kubernetes.
Many data science workflows encompass extract, transform, load (ETL), exploratory data analysis (EDA) steps followed by ML and/or DL follow-on tasks. Arkouda at 50TB+ scale is traditionally run in an HPC environment with the Slurm scheduler. Since ML and DL workflows are increasingly being deployed on Kubernetes in frameworks such as Kubeflow, integration of Arkouda-on-Slurm with Kubernetes is key capability that fosters integration of Arkouda into a variety of workflows deployed on Kubernetes.
Slurm-Bare-Metal-Arkouda-Kubernetes-Integration describes the integration of Arkouda-on-Slurm as well as Arkouda-on-bare-metal with Kubernetes via registration/deregistration of Arkouda with Kubernetes.
Arkouda-Prometheus-Exporter-and-Grafana-Integration describes metrics generation and reporting in Arkouda. A variety of metrics can be generated and exported to Prometheus. The resulting Prometheus metrics can be viewed in Grafana