04 - Project - Reinforcement Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0753acee",
   "metadata": {},
   "source": [
    "# Project: Reinforcement Learning\n",
    "- Bigger field - more learning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9bf2904",
   "metadata": {},
   "source": [
    "### Project field\n",
    "- Use Reinforcement Learning with Q-learning to find solutions to this field.\n",
    "![Field](img/field-2.png \"Field\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00ccac50",
   "metadata": {},
   "source": [
    "### Step 1: Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "917046c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import random"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc68c98d",
   "metadata": {},
   "source": [
    "### Step 2: Create a field\n",
    "\n",
    "![Field](img/field-3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0b6502b",
   "metadata": {},
   "source": [
    "- **__init__**:\n",
    "    - Use a list of list with integer values to represent all the states\n",
    "        - Goal end state should be 1, illegal states -1, other states 0\n",
    "    - Set the state to be random fo the size of states\n",
    "- **done**:\n",
    "    - Check if current state has non-negative values\n",
    "- **get_possible_actions**:\n",
    "    - Set a list to all possible actions **actions = [0, 1, 2, 3]**\n",
    "        - action = 0 is left\n",
    "        - action = 1 is right\n",
    "        - action = 2 is up\n",
    "        - action = 3 is down\n",
    "    - Then check if state is in a position where a possible actions should be removed.\n",
    "    - Finally, return the remaining actions\n",
    "- **update_next_state**:\n",
    "    - Get the current state\n",
    "    - Check if move is illegal, then return current state and -10 in reward\n",
    "    - Otherwise opdate state and return the reward according to new state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc39ab60",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b5b4e69",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe9b8830",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "3bbbe78b",
   "metadata": {},
   "source": [
    "### Step 3: Train the model\n",
    "- Create a $q$-table initialized to all 0\n",
    "    - Use **q_table = np.zeros(...)** *(insert values for ...)*\n",
    "- Set **alpha = .5, gamma = 0.5,** and **epsilon = 0.5**\n",
    "- Create *for*-loop iterating 10000\n",
    "    - Create new field\n",
    "    - While field not done\n",
    "        - Get possible actions and assign to **actions**\n",
    "        - With probability epsilon take a random action, otherwise take the best action\n",
    "            - HINT: **random.uniform(0, 1) < epsilon**\n",
    "            - HINT: Random action: **random.choice(actions)**, and best action: **np.argmax(q_table[field.state])**\n",
    "        - Get current state and assign it to **cur_x, cur_y**\n",
    "        - Update next state and get it and the reward\n",
    "        - Update **q_table[cur_x, cur_y, action] = (1 - alpha)*q_table[cur_x, cur_y, action] + alpha*(reward + gamma*np.max(q_table[next_x, next_y]))**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b9f624f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9cae60b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "5685c738",
   "metadata": {},
   "source": [
    "### Step 4: Solve a task\n",
    "- To see the path make a variable **path = np.zeros((3, 11))**\n",
    "- Create a field **Field()**\n",
    "- To count steps assign **steps = 1**\n",
    "- Assign the start state in the path to **np.nan**.\n",
    "- The we begin: while not solved.\n",
    "    - Get the **action** to take\n",
    "    - Get the next **state**\n",
    "    - Update **path** with **steps**\n",
    "    - Increment **steps** with one\n",
    "- see the **path**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d326a1a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b47755a8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "cd21b45e",
   "metadata": {},
   "source": [
    "> ### Note\n",
    "> - The training phase (Step 3) could just take random actions\n",
    "> - Our example (Step 4) does not learn anything new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc636df1",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}