Continuous_Control_Project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continuous Control\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "\n",
    "### Start the Environment\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The environments corresponding to both versions of the environment are already saved in the Workspace and can be accessed at the file paths provided below.  \n",
    "\n",
    "Please select one of the two options below for loading the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\tgoal_speed -> 1.0\n",
      "\t\tgoal_size -> 5.0\n",
      "Unity brain name: ReacherBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 33\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "# select this option to load version 1 (with a single agent) of the environment\n",
    "#env = UnityEnvironment(file_name='/data/Reacher_One_Linux_NoVis/Reacher_One_Linux_NoVis.x86_64')\n",
    "\n",
    "# select this option to load version 2 (with 20 agents) of the environment\n",
    "env = UnityEnvironment(file_name='Reacher_Windows_x86_64/Reacher.exe')\n",
    "\n",
    "## If this step gives error. restart kernel and clear output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examine the State and Action Spaces\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 20\n",
      "Size of each action: 4\n",
      "There are 20 agents. Each observes a state with length: 33\n",
      "The state for the first agent looks like: [ 0.00000000e+00 -4.00000000e+00  0.00000000e+00  1.00000000e+00\n",
      " -0.00000000e+00 -0.00000000e+00 -4.37113883e-08  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.00000000e+01  0.00000000e+00\n",
      "  1.00000000e+00 -0.00000000e+00 -0.00000000e+00 -4.37113883e-08\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  7.90150833e+00 -1.00000000e+00\n",
      "  1.25147629e+00  0.00000000e+00  1.00000000e+00  0.00000000e+00\n",
      " -5.22214413e-01]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents\n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Take random actions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total score (averaged over agents) this episode: 0.0549999987706542\n",
      "Total score (averaged over agents) this episode: 0.15849999645724894\n",
      "Total score (averaged over agents) this episode: 0.3169999929144979\n"
     ]
    }
   ],
   "source": [
    "#num_agents = 20\n",
    "env_info = env.reset(train_mode=True)[brain_name]      # reset the environment    \n",
    "states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "\n",
    "for _ in range(3):\n",
    "    \n",
    "    while True:\n",
    "        actions = np.random.randn(num_agents, action_size) # select an action (for each agent)\n",
    "        actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1\n",
    "        env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "        next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "        rewards = env_info.rewards                         # get reward (for each agent)\n",
    "        dones = env_info.local_done                        # see if episode finished\n",
    "        scores += env_info.rewards                         # update the score (for each agent)\n",
    "        states = next_states                               # roll over states to next time step\n",
    "        if np.any(dones):                                  # exit loop if episode finished\n",
    "            break\n",
    "    print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "fc1_units = 512 #512  #128 ## 64\n",
    "fc2_units = 256 #256  #64 #32\n",
    "fc3_units = 64\n",
    "#fc4 is 16 by default\n",
    "lr_actor = 1e-4\n",
    "lr_critic = 1e-4\n",
    "buffer_size = int(1e6)\n",
    "batch_size = 128 # 128\n",
    "Gamma = 0.99\n",
    "tau = 0.001\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import importlib\n",
    "import model\n",
    "import Agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = Agent(n_agents = 20,state_size = 33, action_size = 4, random_seed = 2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import deque\n",
    "\n",
    "\n",
    "\n",
    "def cont_control(n_episodes = 1000,  print_every = 100 ):\n",
    "    \n",
    "    scores = []\n",
    "    scores_deque = deque(maxlen=100)\n",
    "    \n",
    "    for i_episode in range(1,n_episodes+1):\n",
    "        \n",
    "        total_r = np.zeros(num_agents)\n",
    "        #total_rewards = 0\n",
    "        #state = env.reset()\n",
    "        env_info = env.reset(train_mode=True)[brain_name]\n",
    "        agent.reset()\n",
    "        statesA = env_info.vector_observations\n",
    "        #state = env_info.vector_observations[0]\n",
    "        #for t in range(max_t):\n",
    "        while True:\n",
    "            \n",
    "            actionsA = agent.act(statesA)\n",
    "            env_info = env.step(actionsA)[brain_name]\n",
    "            next_statesA = env_info.vector_observations\n",
    "            #next_state, reward, done,_ = env.step(action)\n",
    "            rewardsA = env_info.rewards\n",
    "            donesA = env_info.local_done            \n",
    "            \n",
    "            agent.step(statesA, actionsA, rewardsA, next_statesA, donesA)\n",
    "            statesA = next_statesA\n",
    "            total_r += rewardsA\n",
    "            if np.any(donesA):\n",
    "                break\n",
    "        \n",
    "        scores.append(total_r)\n",
    "        scores_deque.append(total_r)\n",
    "        print('\\r Episode:{} \\t Average Score: {:.2f} \\t Average of last 100 episodes: {:.2f}'\\\n",
    "              .format(i_episode, np.mean(scores), np.mean(scores_deque)), end = \"\")\n",
    "        #print(' Episode:{} \\t Average Score: {}'.format(i_episode, np.mean(scores_deque)))\n",
    "        #if i_episode % print_every == 0:\n",
    "            \n",
    "            #print('\\n Episode:{} \\t Average Score: {:.2f}\\n'.format(i_episode, np.mean(scores_deque)), end='')\n",
    "            \n",
    "        if np.mean(scores_deque) >=30:\n",
    "            torch.save(agent.actor_local.state_dict(), 'actor_model.pth')\n",
    "            torch.save(agent.critic_local.state_dict(), 'critic_model.pth')\n",
    "            print('\\n Environment Solved in {} episodes, Average score:{}'.format( (i_episode - 100), np.mean(scores_deque)), end='' )\n",
    "            #torch.save(agent.actor_local.state_dict(), 'actor_model.pth')\n",
    "            #torch.save(agent.critic_local.state_dict(), 'critic_model.pth')\n",
    "            break\n",
    "            \n",
    "    \n",
    "    return scores\n",
    "\n",
    "                \n",
    "                \n",
    "            \n",
    "            \n",
    "            \n",
    "            \n",
    "    \n",
    "    \n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### This step will train your model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "scores = cont_control()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### My model was trained with the output as below"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output above shows the average of last 100 episodes from Episode:5 to Episode:105 which is +30.13"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(scores)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot all 20 agents' scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot( range(1,len(scores)+1), scores)\n",
    "plt.xlabel('# of Episodes')\n",
    "plt.ylabel('Scores')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Please see the readme or the report file for this plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "score_ar = np.asarray(scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "score_ar.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mean of 20 agents' scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "score_agent = np.mean(score_ar, axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "score_agent.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot( range(1,len(score_agent)+1), score_agent)\n",
    "plt.xlabel('# of Episodes')\n",
    "plt.ylabel('Scores')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see the model is quiet stable. The score of +30 was achieved in less than 40 episodes and the algorithm hasn't crashed even after twice the number of episodes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now evaluate our trained Agent's Actor and Critic Models on the 20-agent environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Random actions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env_info = env.reset(train_mode=True)[brain_name]      # reset the environment    \n",
    "states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "while True:\n",
    "    actions = np.random.randn(num_agents, action_size) # select an action (for each agent)\n",
    "    actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1\n",
    "    env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "    next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "    rewards = env_info.rewards                         # get reward (for each agent)\n",
    "    dones = env_info.local_done                        # see if episode finished\n",
    "    scores += env_info.rewards                         # update the score (for each agent)\n",
    "    states = next_states                               # roll over states to next time step\n",
    "    if np.any(dones):                                  # exit loop if episode finished\n",
    "        break\n",
    "print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for episode in range(3):\n",
    "    env_info = env.reset(train_mode=False)[brain_name]        \n",
    "    states = env_info.vector_observations       \n",
    "    score = np.zeros(num_agents)               \n",
    "    \n",
    "    while True:\n",
    "        actions = agent.act(states, add_noise=False)                    \n",
    "        \n",
    "        env_info = env.step(actions)[brain_name]        \n",
    "        next_states = env_info.vector_observations     \n",
    "        rewards = env_info.rewards       \n",
    "        dones = env_info.local_done\n",
    "        score += rewards\n",
    "        states = next_states\n",
    "\n",
    "        if np.any(dones):                              \n",
    "            break\n",
    "\n",
    "    print('Episode: \\t{} \\tScore: \\t{:.2f}'.format(episode, np.mean(score))) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Very impressive and consistent results where in each episode the Agent's score was over 30."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: \t0 \tScore: \t39.11\n",
      "Episode: \t1 \tScore: \t39.08\n",
      "Episode: \t2 \tScore: \t38.89\n"
     ]
    }
   ],
   "source": [
    "agent.actor_local.load_state_dict(torch.load('actor_model.pth',map_location= 'cpu'))\n",
    "agent.critic_local.load_state_dict(torch.load('critic_model.pth', map_location = 'cpu'))\n",
    "\n",
    "\n",
    "for episode in range(3):\n",
    "    env_info = env.reset(train_mode=False)[brain_name]        \n",
    "    states = env_info.vector_observations       \n",
    "    score = np.zeros(num_agents)               \n",
    "    \n",
    "    while True:\n",
    "        actions = agent.act(states, add_noise=False)                    \n",
    "        \n",
    "        env_info = env.step(actions)[brain_name]        \n",
    "        next_states = env_info.vector_observations     \n",
    "        rewards = env_info.rewards       \n",
    "        dones = env_info.local_done\n",
    "        score += rewards\n",
    "        states = next_states\n",
    "\n",
    "        if np.any(dones):                              \n",
    "            break\n",
    "\n",
    "    print('Episode: \\t{} \\tScore: \\t{:.2f}'.format(episode, np.mean(score))) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "drlnd",
   "language": "python",
   "name": "drlnd"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}