lunar_lander_group1_atcheke.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/carlacodes/atchekegroup1lunarlanding/blob/main/lunar_lander_group1_atcheke.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "rKjkh_KgobbF"
   },
   "source": [
    "# Performance Analysis of DQN Algorithm on the Lunar Lander task\n",
    "\n",
    "**By Neuromatch Academy**\n",
    "\n",
    "__Content creators:__ Raghuram Bharadwaj Diddigi, Geraud Nangue Tasse, Yamil Vidal, Sanjukta Krishnagopal, Sara Rajaee\n",
    "\n",
    "__Content editors:__ Spiros Chavlis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "iDR5t137obbP"
   },
   "source": [
    "<p align='center'><img src='https://github.com/NeuromatchAcademy/widgets/blob/master/sponsors.png?raw=True'/></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "brZ9ZJQzobbR"
   },
   "source": [
    "---\n",
    "# Objective, test sentence\n",
    "\n",
    "In this project, the objective is to analyze the performance of the Deep Q-Learning algorithm on an exciting task- Lunar Lander. Before we describe the task, let us focus on two keywords here - analysis and performance. What exactly do we mean by these keywords in the context of Reinforcement Learning (RL)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "wpX-jA7GobbT"
   },
   "source": [
    "---\n",
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "cellView": "form",
    "execution": {},
    "id": "9EanvrrhobbU",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "# @title Install dependencies\n",
    "# !sudo apt-get update > /dev/null 2>&1\n",
    "# !sudo apt-get install -y xvfb python-opengl ffmpeg > /dev/null 2>&1\n",
    "!pip install rarfile --quiet\n",
    "!pip install stable-baselines3[extra] ale-py==0.7.4 --quiet\n",
    "!pip install box2d-py --quiet\n",
    "!pip install gym pyvirtualdisplay --quiet\n",
    "!pip install pyglet --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {},
    "id": "pE3qZJLcobbW",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "# Imports\n",
    "import io\n",
    "import os\n",
    "import glob\n",
    "import torch\n",
    "import base64\n",
    "import stable_baselines3\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from stable_baselines3 import DQN\n",
    "from stable_baselines3.common.results_plotter import ts2xy, load_results\n",
    "from stable_baselines3.common.callbacks import EvalCallback\n",
    "from stable_baselines3.common.env_util import make_atari_env\n",
    "\n",
    "import gym\n",
    "from gym import spaces\n",
    "from gym.wrappers import Monitor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "cellView": "form",
    "execution": {},
    "id": "fp1bUnClobbY",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[WinError 2] The system cannot find the file specified",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m~\\AppData\\Local\\Temp\\ipykernel_2540\\4281111723.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mIPython\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mdisplay\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mipythondisplay\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 6\u001b[1;33m \u001b[0mdisplay\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mDisplay\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mvisible\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0msize\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m1400\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m900\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      7\u001b[0m \u001b[0mdisplay\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstart\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      8\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\atchekegroup1lunarlanding\\lib\\site-packages\\pyvirtualdisplay\\display.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, backend, visible, size, color_depth, bgcolor, use_xauth, retries, extra_args, manage_global_env, **kwargs)\u001b[0m\n\u001b[0;32m     61\u001b[0m             \u001b[0mextra_args\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mextra_args\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     62\u001b[0m             \u001b[0mmanage_global_env\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mmanage_global_env\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 63\u001b[1;33m             \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     64\u001b[0m         )\n\u001b[0;32m     65\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\atchekegroup1lunarlanding\\lib\\site-packages\\pyvirtualdisplay\\xvfb.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, size, color_depth, bgcolor, use_xauth, fbdir, dpi, retries, extra_args, manage_global_env)\u001b[0m\n\u001b[0;32m     48\u001b[0m             \u001b[0mretries\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mretries\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     49\u001b[0m             \u001b[0mextra_args\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mextra_args\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 50\u001b[1;33m             \u001b[0mmanage_global_env\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mmanage_global_env\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     51\u001b[0m         )\n\u001b[0;32m     52\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\atchekegroup1lunarlanding\\lib\\site-packages\\pyvirtualdisplay\\abstractdisplay.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, program, use_xauth, retries, extra_args, manage_global_env)\u001b[0m\n\u001b[0;32m     83\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_retries_current\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     84\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 85\u001b[1;33m         \u001b[0mhelptext\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mget_helptext\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mprogram\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     86\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_has_displayfd\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;34m\"-displayfd\"\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mhelptext\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     87\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_has_displayfd\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\atchekegroup1lunarlanding\\lib\\site-packages\\pyvirtualdisplay\\util.py\u001b[0m in \u001b[0;36mget_helptext\u001b[1;34m(program)\u001b[0m\n\u001b[0;32m     15\u001b[0m         \u001b[0mstdout\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0msubprocess\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mPIPE\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     16\u001b[0m         \u001b[0mstderr\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0msubprocess\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mPIPE\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 17\u001b[1;33m         \u001b[0mshell\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mFalse\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     18\u001b[0m     )\n\u001b[0;32m     19\u001b[0m     \u001b[0m_\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstderr\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcommunicate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\atchekegroup1lunarlanding\\lib\\subprocess.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, encoding, errors, text)\u001b[0m\n\u001b[0;32m    798\u001b[0m                                 \u001b[0mc2pread\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mc2pwrite\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    799\u001b[0m                                 \u001b[0merrread\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0merrwrite\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 800\u001b[1;33m                                 restore_signals, start_new_session)\n\u001b[0m\u001b[0;32m    801\u001b[0m         \u001b[1;32mexcept\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    802\u001b[0m             \u001b[1;31m# Cleanup if the child failed starting.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\atchekegroup1lunarlanding\\lib\\subprocess.py\u001b[0m in \u001b[0;36m_execute_child\u001b[1;34m(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, unused_restore_signals, unused_start_new_session)\u001b[0m\n\u001b[0;32m   1205\u001b[0m                                          \u001b[0menv\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1206\u001b[0m                                          \u001b[0mos\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfspath\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcwd\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0mcwd\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32mNone\u001b[0m \u001b[1;32melse\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1207\u001b[1;33m                                          startupinfo)\n\u001b[0m\u001b[0;32m   1208\u001b[0m             \u001b[1;32mfinally\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1209\u001b[0m                 \u001b[1;31m# Child is launched. Close the parent's copy of those pipe\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mFileNotFoundError\u001b[0m: [WinError 2] The system cannot find the file specified"
     ]
    }
   ],
   "source": [
    "# @title Plotting/Video functions\n",
    "from IPython.display import HTML\n",
    "from pyvirtualdisplay import Display\n",
    "from IPython import display as ipythondisplay\n",
    "\n",
    "display = Display(visible=0, size=(1400, 900))\n",
    "display.start()\n",
    "\n",
    "\"\"\"\n",
    "Utility functions to enable video recording of gym environment\n",
    "and displaying it.\n",
    "To enable video, just do \"env = wrap_env(env)\"\"\n",
    "\"\"\"\n",
    "\n",
    "def show_video():\n",
    "  mp4list = glob.glob('video/*.mp4')\n",
    "  if len(mp4list) > 0:\n",
    "    mp4 = mp4list[0]\n",
    "    video = io.open(mp4, 'r+b').read()\n",
    "    encoded = base64.b64encode(video)\n",
    "    ipythondisplay.display(HTML(data='''<video alt=\"test\" autoplay\n",
    "                loop controls style=\"height: 400px;\">\n",
    "                <source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" />\n",
    "             </video>'''.format(encoded.decode('ascii'))))\n",
    "  else:\n",
    "    print(\"Could not find video\")\n",
    "\n",
    "\n",
    "def wrap_env(env):\n",
    "  env = Monitor(env, './video', force=True)\n",
    "  return env"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "psfdqUCmobba"
   },
   "source": [
    "---\n",
    "# Introduction\n",
    "\n",
    "In a standard RL setting, an agent learns optimal behavior from an environment through a feedback mechanism to maximize a given objective. Many algorithms have been proposed in the RL literature that an agent can apply to learn the optimal behavior. One such popular algorithm is the Deep Q-Network (DQN). This algorithm makes use of deep neural networks to compute optimal actions. In this project, your goal is to understand the effect of the number of neural network layers on the algorithm's performance. The performance of the algorithm can be evaluated through two metrics - Speed and Stability. \n",
    "\n",
    "**Speed:** How fast the algorithm reaches the maximum possible reward. \n",
    "\n",
    "**Stability** In some applications (especially when online learning is involved), along with speed, stability of the algorithm, i.e., minimal fluctuations in performance, is equally important. \n",
    "\n",
    "In this project, you should investigate the following question:\n",
    "\n",
    "**What is the impact of number of neural network layers on speed and stability of the algorithm?**\n",
    "\n",
    "You do not have to write the DQN code from scratch. We have provided a basic implementation of the DQN algorithm. You only have to tune the hyperparameters (neural network size, learning rate, etc), observe the performance, and analyze. More details on this are provided below. \n",
    "\n",
    "Now, let us discuss the RL task we have chosen, i.e., Lunar Lander. This task consists of the lander and a landing pad marked by two flags. The episode starts with the lander moving downwards due to gravity. The objective is to land safely using different engines available on the lander with zero speed on the landing pad as quickly and fuel efficient as possible. Reward for moving from the top of the screen and landing on landing pad with zero speed is between 100 to 140 points. Each leg ground contact yields a reward of 10 points. Firing main engine leads to a reward of -0.3 points in each frame. Firing the side engine leads to a reward of -0.03 points in each frame. An additional reward of -100 or +100 points is received if the lander crashes or comes to rest respectively which also leads to end of the episode. \n",
    "\n",
    "The input state of the Lunar Lander consists of following components:\n",
    "\n",
    "  1. Horizontal Position\n",
    "  2. Vertical Position\n",
    "  3. Horizontal Velocity\n",
    "  4. Vertical Velocity\n",
    "  5. Angle\n",
    "  6. Angular Velocity\n",
    "  7. Left Leg Contact\n",
    "  8. Right Leg Contact\n",
    "\n",
    "The actions of the agents are:\n",
    "  1. Do Nothing\n",
    "  2. Fire Main Engine\n",
    "  3. Fire Left Engine\n",
    "  4. Fire Right Engine\n",
    "\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/NeuromatchAcademy/course-content-dl/main/projects/static/lunar_lander.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "fGCTNteSobbd"
   },
   "source": [
    "---\n",
    "# Basic DQN Implementation\n",
    "\n",
    "We will now implement the DQN algorithm using the existing code base. We encourage you to understand this example and re-use it in an application/project of your choice! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "sEG8U0Pmobbf"
   },
   "source": [
    "Now, let us set some hyperparameters for our algorithm. This is the only part you would play around with, to solve the first part of the project. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {},
    "id": "vJm_qzWkobbg",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "nn_layers = [64,64] #This is the configuration of your neural network. Currently, we have two layers, each consisting of 64 neurons.\n",
    "                    #If you want three layers with 64 neurons each, set the value to [64,64,64] and so on.\n",
    "\n",
    "learning_rate = 0.001 #This is the step-size with which the gradient descent is carried out.\n",
    "                      #Tip: Use smaller step-sizes for larger networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "-9bGzUVPobbh"
   },
   "source": [
    "Now, let us setup our model and the DQN algorithm. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {},
    "id": "1dN91ycLobbh",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "log_dir = \"/tmp/gym/\"\n",
    "os.makedirs(log_dir, exist_ok=True)\n",
    "\n",
    "# Create environment\n",
    "env = gym.make('LunarLander-v2')\n",
    "#You can also load other environments like cartpole, MountainCar, Acrobot. Refer to https://gym.openai.com/docs/ for descriptions.\n",
    "#For example, if you would like to load Cartpole, just replace the above statement with \"env = gym.make('CartPole-v1')\".\n",
    "\n",
    "env = stable_baselines3.common.monitor.Monitor(env, log_dir )\n",
    "\n",
    "callback = EvalCallback(env,log_path = log_dir, deterministic=True) #For evaluating the performance of the agent periodically and logging the results.\n",
    "policy_kwargs = dict(activation_fn=torch.nn.ReLU,\n",
    "                     net_arch=nn_layers)\n",
    "model = DQN(\"MlpPolicy\", env,policy_kwargs = policy_kwargs,\n",
    "            learning_rate=learning_rate,\n",
    "            batch_size=1,  #for simplicity, we are not doing batch update.\n",
    "            buffer_size=1, #size of experience of replay buffer. Set to 1 as batch update is not done\n",
    "            learning_starts=1, #learning starts immediately!\n",
    "            gamma=0.99, #discount facto. range is between 0 and 1.\n",
    "            tau = 1,  #the soft update coefficient for updating the target network\n",
    "            target_update_interval=1, #update the target network immediately.\n",
    "            train_freq=(1,\"step\"), #train the network at every step.\n",
    "            max_grad_norm = 10, #the maximum value for the gradient clipping\n",
    "            exploration_initial_eps = 1, #initial value of random action probability\n",
    "            exploration_fraction = 0.5, #fraction of entire training period over which the exploration rate is reduced\n",
    "            gradient_steps = 1, #number of gradient steps\n",
    "            seed = 1, #seed for the pseudo random generators\n",
    "            verbose=0) #Set verbose to 1 to observe training logs. We encourage you to set the verbose to 1.\n",
    "\n",
    "# You can also experiment with other RL algorithms like A2C, PPO, DDPG etc. Refer to  https://stable-baselines3.readthedocs.io/en/master/guide/examples.html\n",
    "#for documentation. For example, if you would like to run DDPG, just replace \"DQN\" above with \"DDPG\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "0-7OLtdpobbi"
   },
   "source": [
    "Before we train the model, let us look at an instance of Lunar Lander **before training**.  \n",
    "\n",
    "**Note:** The following code for rendering the video is taken from https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_12_01_ai_gym.ipynb#scrollTo=T9RpF49oOsZj"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {},
    "id": "8nEQvdE7obbj",
    "outputId": "27d2e6c0-e8d0-40c3-9373-edcb588cfff5",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'wrap_env' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m~\\AppData\\Local\\Temp\\ipykernel_2540\\3972928859.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mtest_env\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mwrap_env\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mgym\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmake\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"LunarLander-v2\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m \u001b[0mobservation\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtest_env\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreset\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[0mtotal_reward\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;32mwhile\u001b[0m \u001b[1;32mTrue\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m   \u001b[0mtest_env\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'wrap_env' is not defined"
     ]
    }
   ],
   "source": [
    "test_env = wrap_env(gym.make(\"LunarLander-v2\"))\n",
    "observation = test_env.reset()\n",
    "total_reward = 0\n",
    "while True:\n",
    "  test_env.render()\n",
    "  action, states = model.predict(observation, deterministic=True)\n",
    "  observation, reward, done, info = test_env.step(action)\n",
    "  total_reward += reward\n",
    "  if done:\n",
    "    break;\n",
    "\n",
    "# print(total_reward)\n",
    "test_env.close()\n",
    "show_video()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "ZpMABSl9obbk"
   },
   "source": [
    "From the video above, we see that the lander has crashed! \n",
    "It is now the time for training! \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {},
    "id": "I0hzLf1lobbl",
    "outputId": "36cf49ba-e753-476f-9aff-04b9cc3957b9",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "model.learn(total_timesteps=100000, log_interval=10, callback=callback)\n",
    "# The performance of the training will be printed every 10 episodes. Change it to 1, if you wish to\n",
    "# view the performance at every training episode."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "ueR5BIcpobbm"
   },
   "source": [
    "The training takes time. We encourage you to analyze the output logs (set verbose to 1 to print the output logs). The main component of the logs that you should track is \"ep_rew_mean\" (mean of episode rewards). As the training proceeds, the value of \"ep_rew_mean\" should increase. The improvement need not be monotonic, but the trend should be upwards! \n",
    "\n",
    "Along with training, we are also periodically evaluating the performance of the current model during the training. This was reported in logs as follows:\n",
    "\n",
    "```\n",
    "Eval num_timesteps=100000, episode_reward=63.41 +/- 130.02\n",
    "Episode length: 259.80 +/- 47.47\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "ogkoKXiXobbm"
   },
   "source": [
    "Now, let us look at the visual performance of the lander. \n",
    "\n",
    "**Note:** The performance varies across different seeds and runs. This code is not optimized to be stable across all runs and seeds. We hope you will be able to find an optimal configuration! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {},
    "id": "m_WCp8wdobbn",
    "outputId": "931eb7bf-8be5-4131-89c0-4761c0dbef1a",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "env = wrap_env(gym.make(\"LunarLander-v2\"))\n",
    "observation = env.reset()\n",
    "while True:\n",
    "  env.render()\n",
    "  action, _states = model.predict(observation, deterministic=True)\n",
    "  observation, reward, done, info = env.step(action)\n",
    "  if done:\n",
    "    break;\n",
    "\n",
    "env.close()\n",
    "show_video()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "Ov5djtn9obbn"
   },
   "source": [
    "The lander has landed safely!!\n",
    "\n",
    "Let us analyze its performance (speed and stability). For this purpose, we plot the number of time steps on the X-axis and the episodic reward given by the trained model on the Y-axis. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {},
    "id": "asuy5ryIobbo",
    "outputId": "09f10ff3-b1df-42cc-8c10-52f91e581282",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "x, y = ts2xy(load_results(log_dir), 'timesteps')  # Organising the logged results in to a clean format for plotting.\n",
    "plt.plot(x,y)\n",
    "plt.ylim([-300, 300])\n",
    "plt.xlabel('Timesteps')\n",
    "plt.ylabel('Episode Rewards')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "mw3iMGmwobbo"
   },
   "source": [
    "From the above plot, we observe that, although the maximum reward is achieved quickly. Achieving an episodic reward of > 200 is good. We see that the agent has achieved it in less than 50000 timesteps (speed is good!). However, there are a lot of fluctuations in the performance (stability is not good!). \n",
    "\n",
    "Your objective now is to modify the model parameters (nn_layers, learning_rate in the code cell #2 above), run all the cells following it and investigate the stability and speed of the chosen configuration.   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "_bKzapIyobbp"
   },
   "source": [
    "---\n",
    "# Additional Project Ideas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "gzByWGxfobbq"
   },
   "source": [
    "## 1 Play with exploration-exploitation trade-off\n",
    "\n",
    "Exploration (selecting random actions) and exploitation (selecting greedy action) is a crucial component of the DQN algorithm. Explore random actions for a long time will slow down the training process. At the same time, if all actions are not explored enough, it might lead to a sub-optimal performance. In the DQN code above, we have used the following parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {},
    "id": "5TeOBG-Jobbq",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "exploration_initial_eps = 1  # initial value of random action probability. Range is between 0 and 1.\n",
    "exploration_fraction = 0.5  # fraction of entire training period over which the exploration rate is reduced. Range is between 0 and 1.\n",
    "exploration_final_eps = 0.05  # (set by defualt) final value of random action probability. Range is between 0 and 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "dZuGeAULobbq"
   },
   "source": [
    "Your objective is to play around with these parameters and analyze their performance (speed and stability). You can modify these parameters and set them as arguments in DQN(...,exploration_initial_eps = 1, exploration_fraction = 0.5, exploration_final_eps = 0.05,...). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "yGp5JFoAobbr"
   },
   "source": [
    "## 2 Reward Shaping\n",
    "\n",
    "Your objective here is to construct a modified reward function that improves the performance of the Lunar Lander. To this end, you would have to create your own custom environment. An example of a custom environment is given below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {},
    "id": "RStcMxLfobbr",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "# Taken from https://stable-baselines3.readthedocs.io/en/master/guide/custom_env.html\n",
    "class CustomEnv(gym.Env):\n",
    "  \"\"\"Custom Environment that follows gym interface\"\"\"\n",
    "  metadata = {'render.modes': ['human']}\n",
    "\n",
    "  def __init__(self, arg1, arg2):\n",
    "    super(CustomEnv, self).__init__()\n",
    "    # Define action and observation space\n",
    "    # They must be gym.spaces objects\n",
    "    # Example when using discrete actions:\n",
    "    self.action_space = spaces.Discrete(N_DISCRETE_ACTIONS)\n",
    "    # Example for using image as input (channel-first; channel-last also works):\n",
    "    self.observation_space = spaces.Box(low=0, high=255,\n",
    "                                        shape=(N_CHANNELS, HEIGHT, WIDTH), dtype=np.uint8)\n",
    "\n",
    "  def step(self, action):\n",
    "    ...\n",
    "    return observation, reward, done, info\n",
    "  def reset(self):\n",
    "    ...\n",
    "    return observation  # reward, done, info can't be included\n",
    "  def render(self, mode='human'):\n",
    "    ...\n",
    "  def close (self):\n",
    "    ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "RKAA_XL4obbs"
   },
   "source": [
    "As you are only changing the reward structure, you can inherit the original Lunar Lander environment from https://github.com/openai/gym/blob/master/gym/envs/box2d/lunar_lander.py and modify just the \"step\" function. Focus on modifying the following part of the code in the \"step\" function\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {},
    "id": "vW-1yrPyobbs",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "def step(self, actions):\n",
    "  ...\n",
    "  ...\n",
    "  ...\n",
    "  reward = 0\n",
    "  shaping = (\n",
    "      -100 * np.sqrt(state[0] * state[0] + state[1] * state[1])\n",
    "      - 100 * np.sqrt(state[2] * state[2] + state[3] * state[3])\n",
    "      - 100 * abs(state[4])\n",
    "      + 10 * state[6]\n",
    "      + 10 * state[7]\n",
    "  )  # And ten points for legs contact, the idea is if you\n",
    "  # lose contact again after landing, you get negative reward\n",
    "  if self.prev_shaping is not None:\n",
    "      reward = shaping - self.prev_shaping\n",
    "  self.prev_shaping = shaping\n",
    "\n",
    "  reward -= (\n",
    "      m_power * 0.30\n",
    "  )  # less fuel spent is better, about -30 for heuristic landing. You should modify these values.\n",
    "  reward -= s_power * 0.03\n",
    "\n",
    "  done = False\n",
    "  if self.game_over or abs(state[0]) >= 1.0:\n",
    "      done = True\n",
    "      reward = -100\n",
    "  if not self.lander.awake:\n",
    "      done = True\n",
    "      reward = +100\n",
    "  return np.array(state, dtype=np.float32), reward, done, {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "9CCbUKXQobbs"
   },
   "source": [
    "Once you have cutomized your own environment, you can execute that environment by just calling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {},
    "id": "T7IuLBYxobbt",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "#Enter the name of the custome environment you created and uncomment the line below.\n",
    "#env = Custom_LunarLander()\n",
    "# Refer to https://stable-baselines3.readthedocs.io/en/master/guide/custom_env.html, if you would like to create more complex environments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "ZeEbxPsFobbt"
   },
   "source": [
    "## 3 Identify the state information crucial to its performance.\n",
    "\n",
    "Your objective here is to alter the input state information and analyze the performance. The input state of the Lunar Lander consists of following components:\n",
    "\n",
    "  1. Horizontal Position\n",
    "  2. Vertical Position\n",
    "  3. Horizontal Velocity\n",
    "  4. Vertical Velocity\n",
    "  5. Angle\n",
    "  6. Angular Velocity\n",
    "  7. Left Leg Contact\n",
    "  8. Right Leg Contact\n",
    "\n",
    "You can train the algorithm by masking one of the eight components at a time and understand how that affects the performance of the algorithm. Similar to the reward shaping task, you would have to create a custom environment and modify the state space. Again, you can inherit all the necessary functions and modify the following portion of the \"Step\" function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {},
    "id": "e0qQ--Tfobbt",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "def step(self, actions):\n",
    "  ...\n",
    "  ...\n",
    "  ...\n",
    "  state = [ # Remove one component at a time to investigate the effect on performance!\n",
    "            (pos.x - VIEWPORT_W / SCALE / 2) / (VIEWPORT_W / SCALE / 2),\n",
    "            (pos.y - (self.helipad_y + LEG_DOWN / SCALE)) / (VIEWPORT_H / SCALE / 2),\n",
    "            vel.x * (VIEWPORT_W / SCALE / 2) / FPS,\n",
    "            vel.y * (VIEWPORT_H / SCALE / 2) / FPS,\n",
    "            self.lander.angle,\n",
    "            20.0 * self.lander.angularVelocity / FPS,\n",
    "            1.0 if self.legs[0].ground_contact else 0.0,\n",
    "            1.0 if self.legs[1].ground_contact else 0.0,\n",
    "        ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "EIifpb9Kobbu"
   },
   "source": [
    "## 4 Extension to Atari Games\n",
    "\n",
    "In the Lunar Lander task, the input to the algorithm is a vector of state information. Deep RL algorithms can also be applied when the input to the training is image frames, which is the case in the Atari games. For example, consider an Atari game - Pong. In this environment, the observation is an RGB image of the screen, which is an array of shape (210, 160, 3). To train the Pong game, you can start with the following sample code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {},
    "id": "9GEIYsVVobbv",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "## Taken from: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/atari_games.ipynb#scrollTo=f3K4rMXwimBO\n",
    "env = make_atari_env('PongNoFrameskip-v4', n_envs=4, seed=0)\n",
    "\n",
    "##Atari Games take a lot of memory. Following commands crash on Coalb. Run the following code on Colab Pro or your local Jupyter notebook!\n",
    "# env = VecFrameStack(env, n_stack=4)\n",
    "# model = DQN('CnnPolicy', env, verbose=1)  # Note the difference here! We use 'CnnPolicy\" here instead of 'MlpPolicy' as the input is frames.\n",
    "# model.learn(total_timesteps=1) #change the number of timesteps as desired and run this command!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "5I26CZd4obbv"
   },
   "source": [
    "## 5 Obstacle Avoidance and Transfer Learning\n",
    "\n",
    "Your obstacle here is to add an obstacle in the path of the lunar lander (by creating a custom environment as described in point 2 above) and train the model such that the lander lands safely, avoiding collisions. \n",
    "\n",
    "You would first want to devise a mechansim for adding obstacles. For example, you could have an imaginary obstacle at some horizantal and vertical position cooridnates and modify the reward function such that a penalty is levied if the lander comes close to it. \n",
    "\n",
    "An interesting approach to solve this problem is to apply the techniques of transfer learning. For example, you could initialise the neural network model with the weights of the trained model on the original problem to improve the sample effeciency. This can be done using the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {},
    "id": "lSyhxVoLobbw",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "## Specify the load path and uncomment below:\n",
    "\n",
    "# model = load(load_path,\n",
    "#              env=gym.make('LunarLander-v2'),\n",
    "#              custom_objects=None, **kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "4Qy8qSysobbw"
   },
   "source": [
    "Following are some of the resources on transfer learning that you would want to start with. \n",
    "\n",
    "**Research Papers**\n",
    "\n",
    "Surveys:\n",
    "1. (Long, Old, Highly cited) Taylor, M. E.,  et al. (2009). Transfer learning for reinforcement learning domains. https://www.jmlr.org/papers/volume10/taylor09a/taylor09a.pdf\n",
    "\n",
    "2. (Medium, Old, Good for a quick read) Lazaric, A. (2012). Transfer in reinforcement learning: a framework and a survey. https://hal.inria.fr/docs/00/77/26/26/PDF/transfer.pdf\n",
    "\n",
    "3. (Medium, Recent, Good for a quick read) Zhu, Z., Lin, K., & Zhou, J. (2020). Transfer learning in deep reinforcement learning. https://arxiv.org/pdf/2009.07888.pdf\n",
    "\n",
    "4. Specific example:\n",
    "Barreto, A., et al. (2016).  Successor features for transfer in reinforcement learning. https://arxiv.org/pdf/1606.05312"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "vJvZzNrAobbx"
   },
   "source": [
    "## 5(b) Transfer Learning in minigrid environment\n",
    "\n",
    "These are some simple gridworld gym environments designed to be particularly simple, lightweight and fast. Refer to https://github.com/maximecb/gym-minigrid for description of the environments. An example to load a minigrid environment is given below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {},
    "id": "88Bk-iqzobby",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "!pip install gym-minigrid --quiet\n",
    "import gym_minigrid\n",
    "env = gym.make('MiniGrid-Empty-5x5-v0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "lOL5WYR3obby"
   },
   "source": [
    "You can train a standard DQN agent in this env by wrapping the env with full image observation wrappers:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {},
    "id": "PX71Rz3Uobbz",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "env = gym_minigrid.wrappers.ImgObsWrapper(gym_minigrid.wrappers.RGBImgObsWrapper(env))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "o2UiCI5Iobbz"
   },
   "source": [
    "Note that with full image observations, the shape of the image observations may differ between envs. For e.g., MiniGrid-Empty-5x5-v0 is (40,40,3) while MiniGrid-Empty-8x8-v0 is (64,64,3). So you may need to resize the observations for transfer learning to work with the same DQN architecture.\n",
    "\n",
    "Now try training a DQN (or another method) in one (or multiple) minigrid env(s),and see if that knowledge transfers to another (or multiple other) minigrid env(s).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "Qy1rMmg4obbz"
   },
   "source": [
    "## 6 Preference-Based RL (PBRL)\n",
    "\n",
    "PBRL is an exciting sub-area in RL where the traditional reward structure is replaced with human preferences. This setting is very useful in applications where it is difficult to construct a reward function. \n",
    "\n",
    "In the earlier section, we have successfully trained the lunar lander to land safely. Here, the path that the lander follows to land safely can be arbitrary. In this project, using the techniques of PBRL, you will solve the lunar lander problem with an additional requirement that the lander should follow a specially curated path (for example, a straight line path). Following are some of the resources that will help you to get started with this project. \n",
    "\n",
    "**Research papers:**\n",
    "1. Deep Reinforcement Learning\n",
    "from Human Preferences https://papers.nips.cc/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf\n",
    "2. Deep Q-learning from Demonstrations https://arxiv.org/pdf/1704.03732.pdf\n",
    "3. Reward learning from human preferences https://arxiv.org/pdf/1811.06521.pdf\n",
    "4. T-REX https://arxiv.org/pdf/1904.06387.pdf\n",
    "\n",
    "**Code Bases:**\n",
    "1. https://github.com/nottombrown/rl-teacher\n",
    "2. https://github.com/hiwonjoon/ICML2019-TREX\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {},
    "id": "ptpdnpwGobbz"
   },
   "source": [
    "---\n",
    "# References\n",
    "\n",
    "1. Stable Baselines Framework: https://stable-baselines3.readthedocs.io/en/master/guide/examples.html\n",
    "\n",
    "2. Lunar Lander Environment: https://gym.openai.com/envs/LunarLander-v2/\n",
    "\n",
    "3. OpenAI gym environments: https://gym.openai.com/docs/\n",
    "\n",
    "4. A good reference for introduction to RL: http://incompleteideas.net/book/the-book-2nd.html\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "lunar_lander_group1_atcheke",
   "provenance": [],
   "toc_visible": true
  },
  "gpuClass": "standard",
  "kernel": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "kernelspec": {
   "display_name": "PyCharm (atchekegroup1lunarlanding)",
   "language": "python",
   "name": "pycharm-7e003209"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}