Learning Stabilizing Control Policies for a Tensegrity Hopper with Augmented Random Search

Before Learning	After Learning

This repository contains an original source code for the Learning Stabilizing Control Policies for a Tensegrity Hopper with Augmented Random Search paper.

• Augmented Random Search (ARS) implementation is taken from Ray/RLlib
• Tensegrity Simulator is based on NTRTsim from NASA

Table of contents

• Dependencies

• Installation

• Folders structure

• Parameters

• Train it!

• Evaluate it!

• Pretrained Agents

• Citation

• Contact for Issues

• References

Dependencies

• NTRTsim and its dependencies: here

• Python3, Pip3

• Open AI Gym python library

• Ray/RLlib and its dependency (Tensorflow)

• Git

Installation

A. Through installing the specific packages and the libraries:

1. Clone the repository and checkout to the paper branch:

   git clone --branch Learning-Stabilizing-Control-Policies-for-a-Tensegrity-Hopper-with-Augmented-Random-Search-paper https://github.com/hany606/Tensegrity-Robot-IU-Internship19.git
   

2. Go to the repository's directory:

   cd tensegrity-vertical-stability
   

3. Install the dependencies:

   3.1 Install NTRTsim dependencies

   sudo apt-get install g++ libglib2.0-dev curl freeglut3 freeglut3-dev cmake build-essential unzip g++-4.8 python python3-pip

   3.2 Install gym

   pip3 install gym

   3.3 Install Ray/RLlib

   pip3 install ray[rllib]
   pip3 install ray[debug]

4. Running the setup.sh to install NTRTsim

   ./setup.sh

   If any problem appeared using NTRTsim related to g++, check this

   If the setup.sh has failed, try to run it again. (Repeating for three times usually works well)

5. Test the simulator

   a. Run build.sh to build the executable files for the structures

   ./bin/build.sh

   b. Run an example to test the simulator

   ./build/examples/3_prism/AppPrismModel

6. Prepare the environment

   a. Put the following into .bashrc or run it every time to define the home directory for the repository directory

   export TENSEGRITY_HOME="absolute/path/to/the/root/folder"

   For example:

   export TENSEGRITY_HOME="/home/tensegrity-vertical-stability"

   b. To make the command of building the structure faster and easier, put the following into .bashrc to create an alias for the command which is responsible for building the structures executable files:

   alias tensegrityBuild="absolute/path/to/the/bin/folder/build.sh"

   For example:

   alias tensegrityBuild="/home/tensegrity-vertical-stability/bin/build.sh'

7. Install the custom gym environment for Tensegrity

./bin/setup_gym_env.sh

Note: instead of running step 4 and step 7, it is possible just to run:

./bin/setup_helper.sh

B. Using the Docker image for this environment:

docker pull hany606/tensegrity_headless_server:v0.3

Note: the current docker image works with the headless_server branch, which is under development for other experiments, headless_server branch is related to training on servers without GUI.

To work with the docker image that works with the current branch (Paper's branch): it is in "docker" directory

docker build -t tensegrity_img docker/.

Note: to run scripts in headless mode (with xserver display/GUI), it is possible to use Xvfb tool

Example of how to use it:

xvfb-run -a python3 src/dev/gym-tensegrity/gym_tensegrity/envs/jumper_test.py

Or to specify the server number

xvfb-run --server-num=10 python3 src/dev/gym-tensegrity/gym_tensegrity/envs/jumper_test.py

Folders' structure

• bin (NTRTsim): includes the utilities to setup the NTRTsim

• conf (NTRTsim): includes the configuration files that related to the NTRTsim

• build (NTRTsim): includes the executable files for the models after building it using NTRTsim scripts

• env (NTRTsim): includes the libraries that are installed for NTRTsim

• resources (NTRTsim): includes the sources for the libraries for NTRTsim

• src (NTRTsim): includes the source code of tensegrity structures/the paper

  a. dev: includes the source code of the paper, gym environment, and the used tensegrity structure.

  • gym-tensegrity: includes the gym environment inside gym_tensegrity directory

  • learning_scripts: includes the scripts for training and evaluation for this paper

  b. examples (NTRTsim): includes examples for testing the simulator

  c. other (NTRTsim): related to the simulator

Parameters

Parameters are split into two parts. First, a part related to the physical parameters of the tensegrity structure. Second, the part related to the ARS parameters and parameters of the gym_tensegrity environment.

a. Tensegrity Hopper/Jumper Structure and Physical Parameters (In dev/jumper/JumperModel.cpp)

• Density (5kg/cm^3)
• Rod Radius for leg rod and the square side rod (0.20cm)
• Stiffness (3000.0kg/sec^2)
• Damping (30.0kg/sec)
• Pretension (12000.0kg*cm/sec^2)
• Leg Rod length (10.0cm)
• Square Side Rod (10.0cm)
• Max tension (30000kg*cm/sec^2)
• Target Velocity for the actuators (30 cm/sec)

b. gym_tensegrity parameters

• Observation Space: Endpoints of rods (3 * 6); Velocities of the rods' endpoints (3 * 6); Cables' rest lengths (8); 44 dimensions
• Action Space: Delta rest lengths of the cables (continuous); 8 dimensions
• Reward: +1 each time step the structure stays alive without termination
• Termination Condition: The angle between the leg link and the ground should stay in the interval of [-20, 20] degrees, and the angle between the frame link and the ground should stay in the interval of [-40, 40] degrees
• Initial State: Starting from above the ground with height 100cm

c. Augmented Random Search hyperparameters

• Check the training script. For a detailed description of the hyperparameters, see the RLLib documentation.

Train it!

Inside "src/dev/gym-tensegrity/learning_scripts/rllib/training_scripts/", there is a script that has been used for training.

python3 training.py

This is the exact script that was used to obtain the results presented here.

Evaluate it!

Inside "src/dev/gym-tensegrity/learning_scripts/rllib", there is the script which has been used for evaluating the trained model

python3 evaluate.py --agent-path=<path-to-trained-agent> --checkpoint-num=xx

--agent-path: is the path to the trained agent

--checkpoint-num: is the number of the checkpoint to restore the agent in that checkpoint

Example:

python3 evaluate.py --agent-path=trained_agents/train_025_rep_act1_restL/ARS_jumper_29224120_2020-01-21_19-35-20j61sj43o/ --checkpoint-num=605

Or for different models checkpoints and different configuration files

python3 evaluate.py --evaluation-file=<path-to-trained-agent-training-checkpoint-xx> --agent-config-file=<path-to-config-file-for-trained-agent.json>

--evaluation-file: is the path to the checkpoint for the trained agent that will be used to restore the trained model to be evaluated

--agent-config-file: is the path to the json file which includes all the configurations and parameters to the trained agent

Example:

python3 evaluate.py --evaluation-file=trained_agents/train_025_rep_act1_restL/ARS_jumper_29224120_2020-01-21_19-35-20j61sj43o/checkpoint_60/checkpoint-60 --agent-config-file=trained_agents/train_025_rep_act1_restL/ARS_jumper_29224120_2020-01-21_19-35-20j61sj43o/params.json

Pretrained Agents

Inside "src/dev/gym-tensegrity/learning_scripts/rllib/trained_agents", there is the successful agent that has been trained with the checkpoints, tfevents file, the parameters for the training and the progress log

• train_025_rep_act1_restL/ARS_jumper_29224120_2020-01-21_19-35-20j61sj43o: The configuration of the gym_tensegrity and the parameters for ARS are included in params.json

Citation

TODO: https://arxiv.org/pdf/2004.02641.pdf (@inproceedings/@conference ??)

Contact for Issues

Hany Hamed: h.hamed.elanwar@gmail.com / h.hamed@innopolis.university

Vlad Kurenkov: v.kurenkov@innopolis.ru

References

• NASA Tensegrity Robotics Toolkit Simulator (NTRTsim)

• Ray/RLlib

• ML Code Completeness Checklist