Lightweight code for "Goal-directed graph construction using reinforcement learning"

NOTE. This is a lightweight version of the code for this paper with the GPU dependency stripped down, which makes it easier to run the code in case you don't have a GPU. Also useful if you just want to try out the algorithm. The full version of the code is at this repository, which enables reproducing the results in the paper.

This is the code for the article Goal-directed graph construction using reinforcement learning by Victor-Alexandru Darvariu, Stephen Hailes and Mirco Musolesi, Proc. R. Soc. A. 477:20210168. If you use this code, please consider citing:

@article{darvariu2021goal, 
  author = {Darvariu, Victor-Alexandru and Hailes, Stephen and Musolesi, Mirco}, 
  title = {Goal-directed graph construction using reinforcement learning}, 
  journal = {Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences},
  year = 2021, 
  month = {oct}, 
  publisher = {The Royal Society Publishing}, 
  volume = {477}, 
  number = {2254}, 
  pages={20210168},
  doi = {10.1098/rspa.2021.0168}, 
  url = {https://doi.org/10.1098/rspa.2021.0168}, 
}

License

MIT.

Prerequisites

Please ensure that you clone this repository under the relnet root directory, e.g.

git clone git@github.com:VictorDarvariu/graph-construction-rl-lite.git relnet

Currently tested on Linux and MacOS (specifically, CentOS 7.4.1708 and Mac OS Big Sur 11.2.3), can also be adapted to Windows through WSL. Makes heavy use of Docker, see e.g. here for how to install on CentOS. Tested with Docker 19.03. The use of Docker largely does away with dependency and setup headaches, making it significantly easier to reproduce the reported results.

Configuration

The Docker setup uses Unix groups to control permissions. You can reuse an existing group that you are a member of, or create a new group groupadd -g GID GNAME and add your user to it usermod -a -G GNAME MYUSERNAME.

Create a file relnet.env at the root of the project (see relnet_example.env) and adjust the paths within: this is where some data generated by the container will be stored. Also specify the group ID and name created / selected above.

Add the following lines to your .bashrc, replacing /home/john/git/relnet with the path where the repository is cloned.

export RN_SOURCE_DIR='/home/john/git/relnet'
set -a
. $RN_SOURCE_DIR/relnet.env
set +a

export PATH=$PATH:$RN_SOURCE_DIR/scripts

Make the scripts executable (e.g. chmod u+x scripts/*) the first time after cloning the repository, and run apply_permissions.sh in order to create and permission the necessary directories.

Managing the container

Some scripts are provided for convenience. To build the container (note, this will take a significant amount of time e.g. 2 hours, as some packages are built from source):

update_container.sh

To start it:

manage_container.sh up

To stop it:

manage_container.sh stop

To restart the container:

restart.sh

Compiling objective function extension

(First-time setup only) with the container running (via manage_container.sh up above), execute the following command:

docker exec -it relnet-manager /bin/bash -c "cd /relnet/relnet/objective_functions && make"

Setting up synthetic graph data

Synthetic data will be automatically generated when the experiments are ran and stored to $RN_EXPERIMENT_DIR/stored_graphs.

Accessing the services

There are several services running on the manager node.

• Jupyter notebook server: http://localhost:8888 (make sure to select the python-relnet kernel which has the appropriate dependencies)
• Tensorboard (currently disabled due to its large memory footprint): http://localhost:6006

The first time Jupyter is accessed it will prompt for a token to enable password configuration, it can be grabbed by running docker exec -it relnet-manager /bin/bash -c "jupyter notebook list".

Running experiments

The file relnet/experiment_launchers/run_rnet_dqn.py contains the configuration to run the RNet-DQN algorithm on synthetic graphs. You may modify objective functions, hyperparameters etc. to suit your needs. Example for how to run:

docker exec -it relnet-manager /bin/bash -c "source activate ucfadar-relnet && python relnet/experiment_launchers/run_rnet_dqn.py"

Problems with jupyter kernel

In case the python-relnet kernel is not found, try reinstalling the kernel by running docker exec -it relnet-manager /bin/bash -c "source activate ucfadar-relnet; python -m ipykernel install --user --name relnet --display-name python-relnet"

Contact

If you face any issues or have any queries feel free to contact v.darvariu@ucl.ac.uk and I will be happy to assist.