New PA 3 startup files #1

SumanthRH · 2024-03-05T21:43:40Z

Updated setup for PA 3 for DSC 204A.

Changes:

For PA2, we had to downgrade numpy and pandas versions for compatibility with pygloo. This ended up breaking some installations like modin. We no longer need this. Container environments should be more consistent now.
Higher cpu and memory guarantees for the workers. Modin does eager execution which blows up memory consumption.
Added public and /datasets mount for all the nodes.
Fixed ray start commands for object store memory and total memory args. This is still hardcoded for the worker nodes as of now.

…p-login

agt and others added 16 commits February 23, 2024 14:49

Updating start-workers to support /public mounts, launch-sh from dsml…

ead67e1

…p-login

propagate cpu/mem default changes from prior version

ac5df17

Update ray start in start-workers.sh

4bcb00b

Update ray-ml container version

7bc2013

Update docker.yml to trigger builds with all edits

2380c15

updated dockerfile

139f9a0

added /datasets to workers

3415832

Update start-cluster.sh for cpu_guarantee

af846b9

Update start-workers.sh for docker image path

40bafaa

Update start-workers.sh

9d441c9

add userscript support for start-workers.sh

1bb5b64

changed version to 2.9.1

c285064

add private dir mount

5cc1ab3

private and public mounts work for workers

6e6d42e

add memory limits for ray in head node

d149492

update for limits and testing gensim

daa0746

SumanthRH merged commit 909f10f into main Mar 5, 2024
2 checks passed

SumanthRH deleted the pa3 branch March 5, 2024 21:51

Provide feedback