week01_intro/seminar_gymnasium_interface.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys, os\n",
    "if 'google.colab' in sys.modules and not os.path.exists('.setup_complete'):\n",
    "    !wget -q https://raw.githubusercontent.com/yandexdataschool/Practical_RL/master/setup_colab.sh -O- | bash\n",
    "\n",
    "    !touch .setup_complete\n",
    "\n",
    "# This code creates a virtual display to draw game images on.\n",
    "# It will have no effect if your machine has a monitor.\n",
    "if type(os.environ.get(\"DISPLAY\")) is not str or len(os.environ.get(\"DISPLAY\")) == 0:\n",
    "    !bash ../xvfb start\n",
    "    os.environ['DISPLAY'] = ':1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OpenAI Gym --> Farama Gymnasium\n",
    "\n",
    "We're gonna spend several next weeks learning algorithms that solve decision processes. We are then in need of some interesting decision problems to test our algorithms.\n",
    "\n",
    "That's where Gymnasium comes into play. It's a Python library that wraps many classical decision problems including robot control, videogames and board games.\n",
    "\n",
    "The library Gym by OpenAi has been replaced by Gymnsasium while saving all functionality comparable with the latest version of Gym.\n",
    "\n",
    "Announce: https://farama.org/Announcing-The-Farama-Foundation\n",
    "\n",
    "Github: https://github.com/Farama-Foundation/Gymnasium\n",
    "\n",
    "Documentation: https://gymnasium.farama.org/\n",
    "\n",
    "So here's how it works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gymnasium as gym\n",
    "\n",
    "env = gym.make(\"MountainCar-v0\", render_mode=\"rgb_array\")\n",
    "env.reset()\n",
    "\n",
    "plt.imshow(env.render())\n",
    "print(\"Observation space:\", env.observation_space)\n",
    "print(\"Action space:\", env.action_space)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: if you're running this on your local machine, you'll see a window pop up with the image above. Don't close it, just alt-tab away."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gymnasium interface\n",
    "\n",
    "The three main methods of an environment are\n",
    "* `reset()`: reset environment to the initial state, return first observation and dict with auxiliary info\n",
    "* `render()`: show current environment state (a more colorful version :) )\n",
    "* `step(a)`: commit action `a` and return `(new_observation, reward, terminated, truncated, info)`\n",
    " * `new_observation`: an observation right after committing the action `a`\n",
    " * `reward`: a number representing your reward for committing action `a`\n",
    " * `terminated`: True if the MDP has just finished, False if still in progress\n",
    " * `truncated`: True if the number of steps elapsed >= max episode steps\n",
    " * `info`: some auxiliary stuff about what just happened. For now, ignore it.\n",
    "\n",
    "A detailed explanation of the difference between `terminated` and `truncated` and how it should be used:\n",
    "1. https://gymnasium.farama.org/tutorials/gymnasium_basics/handling_time_limits/\n",
    "2. https://gymnasium.farama.org/content/migration-guide/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set seed to reproduce initial state in stochastic environment\n",
    "obs0, info = env.reset(seed=0)\n",
    "print(\"initial observation code:\", obs0)\n",
    "\n",
    "obs0, info = env.reset(seed=1)\n",
    "print(\"initial observation code:\", obs0)\n",
    "\n",
    "# Note: in MountainCar, observation is just two numbers: car position and velocity\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"taking action 2 (right)\")\n",
    "new_obs, reward, terminated, truncated, _ = env.step(2)\n",
    "\n",
    "print(\"new observation code:\", new_obs)\n",
    "print(\"reward:\", reward)\n",
    "print(\"is game over?:\", terminated)\n",
    "print(\"is game truncated due to time limit?:\", truncated)\n",
    "\n",
    "# Note: as you can see, the car has moved to the right slightly (around 0.0005)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Play with it\n",
    "\n",
    "Below is the code that drives the car to the right. However, if you simply use the default policy, the car will not reach the flag at the far right due to gravity.\n",
    "\n",
    "__Your task__ is to fix it. Find a strategy that reaches the flag. \n",
    "\n",
    "You are not required to build any sophisticated algorithms for now, and you definitely don't need to know any reinforcement learning for this. Feel free to hard-code :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython import display\n",
    "\n",
    "# Create env manually to set time limit. Please don't change this.\n",
    "TIME_LIMIT = 250\n",
    "env = gym.wrappers.TimeLimit(\n",
    "    gym.make(\"MountainCar-v0\", render_mode=\"rgb_array\"),\n",
    "    max_episode_steps=TIME_LIMIT + 1,\n",
    ")\n",
    "actions = {\"left\": 0, \"stop\": 1, \"right\": 2}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def policy(obs, t):\n",
    "    # Write the code for your policy here. You can use the observation\n",
    "    # (a tuple of position and velocity), the current time step, or both,\n",
    "    # if you want.\n",
    "    position, velocity = obs\n",
    "\n",
    "    # This is an example policy. You can try running it, but it will not work.\n",
    "    # Your goal is to fix that. You don't need anything sophisticated here,\n",
    "    # and you can hard-code any policy that seems to work.\n",
    "    # Hint: think how you would make a swing go farther and faster.\n",
    "    return actions[\"right\"]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4, 3))\n",
    "display.clear_output(wait=True)\n",
    "\n",
    "obs, _ = env.reset()\n",
    "for t in range(TIME_LIMIT):\n",
    "    plt.gca().clear()\n",
    "\n",
    "    action = policy(obs, t)  # Call your policy\n",
    "    obs, reward, terminated, truncated, _ = env.step(\n",
    "        action\n",
    "    )  # Pass the action chosen by the policy to the environment\n",
    "\n",
    "    # We don't do anything with reward here because MountainCar is a very simple environment,\n",
    "    # and reward is a constant -1. Therefore, your goal is to end the episode as quickly as possible.\n",
    "\n",
    "    # Draw game image on display.\n",
    "    plt.imshow(env.render())\n",
    "\n",
    "    display.display(plt.gcf())\n",
    "    display.clear_output(wait=True)\n",
    "\n",
    "    if terminated or truncated:\n",
    "        print(\"Well done!\")\n",
    "        break\n",
    "else:\n",
    "    print(\"Time limit exceeded. Try again.\")\n",
    "\n",
    "display.clear_output(wait=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert obs[0] > 0.47\n",
    "print(\"You solved it!\")\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}