distributedRL is a framework for prototyping disrtibuted reinforcement learning using Pytorch, Ray, and ZeroMQ (and more). You can make use of the structural components to easily distribute your reinforcement learning algorithm, on a single-machine.
Currently, an implementation of Ape-X DQN is available. IMPALA and SEED are planned to be implemented.
Feel free to reach out (cjy2129 at columbia dot edu) or raise an issue if you have any questions!
ApeXLearnerandApeXWorkerare implemented as abstract classes (that inheritcommon.abstract.Learnerandcommon.abstract.Workerrespectively). To extend Ape-X to any off-policy RL algorithm, you just have to implement algorithm-specific details, such as action selection, environment step, and learning step (check the abstract classes for more detail).- I use ZeroMQ for inter-process communication instead of the Ray's built-in features. (1) I wanted the data-passing mechanism and serialization to be a bit more explicit, and (2) ray's inter-process communication forbids calling remote operations, which I found to be a bit restricting for this purpose.
clone the repository, then
conda env create -f environment.yml
conda activate distrl
pip install -e .
To be added
To be added
I thank @Avrech for his helpful discussion, suggestions and enhancements. In particular, on identifying bugs and bottlenecks and improving asynchrony of the Ape-X implementation.