Soccer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Collaboration and Competition\n",
    "\n",
    "---\n",
    "\n",
    "Congratulations for completing the third project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893) program!  In this notebook, you will learn how to control agents in a more challenging environment, where the goal is to train a team of agents to play soccer.  **Note that this exercise is optional!**\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing the necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np\n",
    "import torch\n",
    "import random\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "import sys\n",
    "import os\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Soccer.app\"`\n",
    "- **Windows** (x86): `\"path/to/Soccer_Windows_x86/Soccer.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Soccer_Windows_x86_64/Soccer.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Soccer_Linux/Soccer.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Soccer_Linux/Soccer.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Soccer_Linux_NoVis/Soccer.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Soccer_Linux_NoVis/Soccer.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Soccer.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Soccer.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env = UnityEnvironment(file_name=\"...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we obtain separate brains for the striker and goalie agents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the brain names\n",
    "print(env.brain_names)\n",
    "\n",
    "# set the goalie brain\n",
    "g_brain_name = env.brain_names[0]\n",
    "g_brain = env.brains[g_brain_name]\n",
    "\n",
    "# set the striker brain\n",
    "s_brain_name = env.brain_names[1]\n",
    "s_brain = env.brains[s_brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)\n",
    "\n",
    "# number of agents \n",
    "num_g_agents = len(env_info[g_brain_name].agents)\n",
    "print('Number of goalie agents:', num_g_agents)\n",
    "num_s_agents = len(env_info[s_brain_name].agents)\n",
    "print('Number of striker agents:', num_s_agents)\n",
    "\n",
    "# number of actions\n",
    "g_action_size = g_brain.vector_action_space_size\n",
    "print('Number of goalie actions:', g_action_size)\n",
    "s_action_size = s_brain.vector_action_space_size\n",
    "print('Number of striker actions:', s_action_size)\n",
    "\n",
    "# examine the state space \n",
    "g_states = env_info[g_brain_name].vector_observations\n",
    "g_state_size = g_states.shape[1]\n",
    "print('There are {} goalie agents. Each receives a state with length: {}'.format(g_states.shape[0], g_state_size))\n",
    "s_states = env_info[s_brain_name].vector_observations\n",
    "s_state_size = s_states.shape[1]\n",
    "print('There are {} striker agents. Each receives a state with length: {}'.format(s_states.shape[0], s_state_size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you will learn how to use the Python API to control the agents and receive feedback from the environment.\n",
    "\n",
    "Once this cell is executed, you will watch the agents' performance, if they select actions at random with each time step.  A window should pop up that allows you to observe the agents.\n",
    "\n",
    "Of course, as part of the project, you'll have to change the code so that the agents are able to use their experiences to gradually choose better actions when interacting with the environment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(2):                                         # play game for 2 episodes\n",
    "    env_info = env.reset(train_mode=False)                 # reset the environment    \n",
    "    g_states = env_info[g_brain_name].vector_observations  # get initial state (goalies)\n",
    "    s_states = env_info[s_brain_name].vector_observations  # get initial state (strikers)\n",
    "    g_scores = np.zeros(num_g_agents)                      # initialize the score (goalies)\n",
    "    s_scores = np.zeros(num_s_agents)                      # initialize the score (strikers)\n",
    "    while True:\n",
    "        # select actions and send to environment\n",
    "        g_actions = np.random.randint(g_action_size, size=num_g_agents)\n",
    "        s_actions = np.random.randint(s_action_size, size=num_s_agents)\n",
    "        actions = dict(zip([g_brain_name, s_brain_name], \n",
    "                           [g_actions, s_actions]))\n",
    "        env_info = env.step(actions)                       \n",
    "        \n",
    "        # get next states\n",
    "        g_next_states = env_info[g_brain_name].vector_observations         \n",
    "        s_next_states = env_info[s_brain_name].vector_observations\n",
    "        \n",
    "        # get reward and update scores\n",
    "        g_rewards = env_info[g_brain_name].rewards  \n",
    "        s_rewards = env_info[s_brain_name].rewards\n",
    "        g_scores += g_rewards\n",
    "        s_scores += s_rewards\n",
    "        \n",
    "        # check if episode finished\n",
    "        done = np.any(env_info[g_brain_name].local_done)  \n",
    "        \n",
    "        # roll over states to next time step\n",
    "        g_states = g_next_states\n",
    "        s_states = s_next_states\n",
    "        \n",
    "        # exit loop if episode finished\n",
    "        if done:                                           \n",
    "            break\n",
    "    print('Scores from episode {}: {} (goalies), {} (strikers)'.format(i+1, g_scores, s_scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. It's Your Turn!\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}