wrnchAI_Hands_on.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "# wrnchAI Hand's on\n",
    "\n",
    "## Objectives\n",
    "\n",
    "At the end of this presentation you should be able to install and use wrnchAI\n",
    "\n",
    "- What's wrnch?\n",
    "    - Live example\n",
    "    - Introduction to pose estimation\n",
    "    - Joints definition\n",
    "\n",
    "- Getting started with:\n",
    "    - Cloud REST API\n",
    "    - Edge SDK\n",
    "\n",
    "This getting started notebook will be shared to all participants"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Whats wrnch?\n",
    "\n",
    "## Human pose estimation\n",
    "- 2d, 3d, head and hand pose estimations\n",
    "- Activity recognition (with Edge SDK)\n",
    "\n",
    "## How to use wrnch\n",
    "- Cloud: REST API or Python wrapper\n",
    "- Edge: `Python` API or `C++` API "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "---\n",
    "# Getting started with joint definition "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 1: Register and test wrnchAI\n",
    "### Register you developer account\n",
    "\n",
    "- **You MUST register through this link to get a 5 days free dev license: https://devportal.wrnch.ai/mcwics_winter2020**\n",
    "- NB: if you don't use this URL, you **won't** receive a free license\n",
    "    \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Submit an image or a video\n",
    "- For this demo we will use this image:\n",
    "\n",
    "\n",
    "![test-image](img/small/test-image.jpg)\n",
    "\n",
    "\n",
    "- Once register you can submit video to our test page to this address:\n",
    "    https://devportal.wrnch.ai/wrnchcloud\n",
    "\n",
    "- Ask for both JSON and Annotated media with those options:\n",
    "    - Head Pose Estimation\n",
    "    - Hand Pose Estimation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 2: Retrieve results\n",
    "\n",
    "Once our servers have processed the image, download the results. If you have selected more than one worktype, you will receive them as a zip. In our example the zip has an annotated_media and a json file\n",
    "\n",
    "The annotated media will look like this:\n",
    "\n",
    "![result img](img/small/test-image-processed.jpeg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 3. Process the results\n",
    "\n",
    "## Our goal\n",
    "-> Lets verify that his left wrist is on his left knee\n",
    "\n",
    "### 1. Import the json file\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'frame_time': 0, 'height': 819, 'persons': [{'hand_pose': {'left': {'bbox': {'height': 0.2857142984867096, 'minX': 0.6681647300720215, 'minY': 0.3588320314884186, 'width': 0.29249998927116394}, 'is_main': False, 'joints': [0.7872003316879272, 0.49528399109840393, 0.7665179371833801, 0.510883808135\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "def json_r(filename):\n",
    "   with open(filename) as f_in:\n",
    "       return(json.load(f_in))\n",
    "\n",
    "pose_estimation = json_r('test-image-processed.json')\n",
    "\n",
    "print(str(pose_estimation['frames'])[:300])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['hand_pose', 'head_pose', 'id', 'pose2d'])\n"
     ]
    }
   ],
   "source": [
    "man_pose = pose_estimation['frames'][0]['persons'][0]\n",
    "\n",
    "print(man_pose.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.3711889684200287,\n",
      " 0.7936080694198608,\n",
      " 0.5704107880592346,\n",
      " 0.7095877528190613,\n",
      " 0.3788127601146698,\n",
      " 0.6029024720191956,\n",
      " 0.5351067781448364,\n",
      " 0.5762162208557129,\n",
      " 0.7694329023361206,\n",
      " 0.5570101141929626]\n"
     ]
    }
   ],
   "source": [
    "import pprint\n",
    "# Lets print the 10 first joint coordinates\n",
    "pprint.pprint(man_pose['pose2d']['joints'][0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### 2. Interpret the coordinates\n",
    " \n",
    " Each joint has a 2d coordinate in this space: \n",
    " \n",
    " ![joint coor](img/small/with-arrow.png)\n",
    " \n",
    " #### We define both 2d and 3d coordinate on this page: https://devportal.wrnch.ai/wrnchai_sdk/coord_spaces"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Joints definition\n",
    "#### For more information and figure, refer to our joint definition documentation: \n",
    "https://devportal.wrnch.ai/wrnchai_sdk/joint_definitions\n",
    "\n",
    "\n",
    "![hands joint doc](img/small/j25_body_joints.jpg)\n",
    "\n",
    "\n",
    "#### There are 25 joints and the joints json hold 50 values (25 pairs of coordinates).\n",
    "Joint 0 coordinates, (x,y) = (man_pose['pose2d']['joints'][0],man_pose['pose2d']['joints'][1])\n",
    "<br>\n",
    "Joint n coordinates, (x,y) = (man_pose['pose2d']['joints'][2n],man_pose['pose2d']['joints'][2n+1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50\n"
     ]
    }
   ],
   "source": [
    "print(len(man_pose['pose2d']['joints']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Lets determine if his left wrist is over his left knee.\n",
    "We should have :\n",
    "\n",
    "`xLWRIST ~= xLKNEE`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7735595107078552\n",
      "0.7694329023361206\n"
     ]
    }
   ],
   "source": [
    "xLWRIST = man_pose['pose2d']['joints'][30]\n",
    "xLKNEE = man_pose['pose2d']['joints'][8]\n",
    "\n",
    "print(xLWRIST)\n",
    "print(xLKNEE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "-> Indeed the wrist and the elbow are close in the frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Getting started with the cloud API\n",
    "All of this is also documented here: https://devportal.wrnch.ai/wrnchcloud/api_docs#introduction\n",
    "<br>\n",
    "Here is an example of using REST API in python.\n",
    "<br>\n",
    "If you would like to run the code below in your environment, please make sure that your python version is at least 3.6 and you have installed all the listed dependencies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "First install requests. The python http client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: requests in /home/pierre/.pyenv/versions/3.7.3/lib/python3.7/site-packages (2.20.1)\n",
      "Requirement already satisfied: idna<2.8,>=2.5 in /home/pierre/.pyenv/versions/3.7.3/lib/python3.7/site-packages (from requests) (2.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/pierre/.pyenv/versions/3.7.3/lib/python3.7/site-packages (from requests) (2019.3.9)\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/pierre/.pyenv/versions/3.7.3/lib/python3.7/site-packages (from requests) (3.0.4)\n",
      "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /home/pierre/.pyenv/versions/3.7.3/lib/python3.7/site-packages (from requests) (1.23)\n",
      "\u001b[33mYou are using pip version 19.0.3, however version 19.3.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install requests\n",
    "import requests\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The two paths we will need for this example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "LOGIN_URL = 'https://api.wrnch.ai/v1/login'\n",
    "JOBS_URL = 'https://api.wrnch.ai/v1/jobs'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "1. Retrieve the cloud API key\n",
    "\n",
    "Click on the key icon on this page: https://devportal.wrnch.ai/licenses to find your cloud API key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#Save the Cloud API key for next cell to work\n",
    "API_KEY = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "2. Authenticate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"access_token\": \"eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOjc1NiwibGljZW5zZV9pZCI6NTkxLCJhZGRvbnMiOnt9LCJleHAiOjE1NzM4MzkyNjMsImlkZW50aXR5Ijo3NTYsInJvbGUiOiJpbmRfY2xpZW50IiwiaWF0IjoxNTczODMyMDYzLCJqdGkiOiJhNzYyYzBiYWVmZmVmMmUzY2Y1ZTllMmM5Y2YxNjdhNDE5M2I5YzcxMDBiZTUyNDhmMzcyZDBiNzQxNWVjNDMwIiwidHlwZSI6ImFjY2VzcyIsImZyZXNoIjoiZmFmIn0.OOWitopW0bsZ0Ne5hLsyhh3PzctKvBm_HLwUeYYXLQ4\"}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "resp_auth = requests.post(LOGIN_URL,data={'api_key':API_KEY})\n",
    "print(resp_auth.text)\n",
    "# the jwt token is valid for an hour\n",
    "JWT_TOKEN = json.loads(resp_auth.text)['access_token']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Now lets submit a file for processing\n",
    "1. Open the file as a byte stream\n",
    "2. Send a post request with authentification and the file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "with open('img/test-image.jpg', 'rb') as f:\n",
    "    resp_sub_job = requests.post(JOBS_URL,\n",
    "                                 headers={'Authorization':f'Bearer {JWT_TOKEN}'},\n",
    "                                 files={'media':f},\n",
    "                                 data={'work_type':'json'}\n",
    "                                )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Status code: 202\n",
      "Response: {\"job_id\": \"1cc5c026-1ac8-483b-8ff5-83ec09fa425e\"}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "job_id = json.loads(resp_sub_job.text)['job_id']\n",
    "print('Status code:',resp_sub_job.status_code)\n",
    "print('Response:',resp_sub_job.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The status code should be 202 and return a job_id.\n",
    "\n",
    "We should wait a few seconds to retrieve the result. You could check the job status in this link https://devportal.wrnch.ai/jobs OR using following command\n",
    "\n",
    "GET_JOB_STATUS_URL = 'https://api.wrnch.ai/v1/status' + '/' + job_id\n",
    "response = requests.get(GET_JOB_STATUS_URL, headers={'Authorization':f'Bearer {JWT_TOKEN}'})\n",
    "print('Job status:', response.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Now lets retrieve the result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://api.wrnch.ai/v1/jobs/1cc5c026-1ac8-483b-8ff5-83ec09fa425e\n",
      "Status code: 200\n",
      "\n",
      "Response: {\"file_info\":{\"joint_definitions\":{\"hands\":\"hand21\",\"head\":\"wrFace20\",\"pose2d\":\"j25\",\"pose3d_ik\":\"extended\",\"pose3d_raw\":\"j25\"}},\"frames\":[{\"frame_time\":0,\"height\":614,\"persons\":[{\"id\":0,\"pose2d\":{\"bbox\":{\"height\":0.9895586371421814,\"minX\":0.2897970974445343,\"minY\":0.039630815386772156,\"width\":0.5532262325286865},\"is_main\":true,\"joints\":[0.37112537026405334,0.7902586460113525,0.5703703165054321,0.7099169492721558,0.3749939203262329,0.6030898690223694,0.5311734676361084,0.5763961672782898,0.7657291889190674,0.5610739588737488,0.6757756471633911,0.866474986076355,0.45308369398117065,0.5897430181503296,0.5196003913879395,0.2996460795402527,0.5196273922920227,0.29780837893486023,0.5455899834632874,0.13356269896030426,0.5350296497344971,0.5954576730728149,0.3633612096309662,0.46940380334854126,0.42190080881118774,0.28249606490135193,0.6172999143600464,0.31679609417915344,0.6875200271606445,0.4581219255924225,0.7735621333122253,0.4886617064476013,0.5779774188995361,0.13753801584243774,0.5470162034034729,0.1220940351486206,0.49219584465026855,0.14887282252311707,0.5975579619407654,0.12972120940685272,-0.00390625,-0.0038171824999153614,0.39449259638786316,0.908473789691925,0.7969211339950562,0.9467262029647827,0.335899293422699,0.7709774374961853,0.6288465857505798,0.9007530212402344],\"scores\":[0.8040049076080322,0.9206298589706421,0.6534831523895264,0.6234648823738098,0.8405327200889587,0.895474910736084,0.6384739875793457,0.8672735691070557,0.8769335746765137,0.9407069087028503,0.8741081357002258,0.9001782536506653,0.8620328903198242,0.8725142478942871,0.791862428188324,0.8561787605285645,0.9511832594871521,0.961635947227478,0.9231566190719604,0.9373281002044678,0.0,0.8867651224136353,0.9401049017906189,0.8837882280349731,0.8658027648925781]}}],\"width\":600}]}\n"
     ]
    }
   ],
   "source": [
    "GET_JOB_URL = JOBS_URL + '/' +job_id\n",
    "print(GET_JOB_URL)\n",
    "resp_get_job = requests.get(GET_JOB_URL,headers={'Authorization':f'Bearer {JWT_TOKEN}'})\n",
    "print('Status code:',resp_get_job.status_code)\n",
    "print('\\nResponse:',resp_get_job.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Lets see if the left wrist and left knee are touching:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7735621333122253\n",
      "0.7657291889190674\n"
     ]
    }
   ],
   "source": [
    "cloud_pose_estimation = json.loads(resp_get_job.text)\n",
    "cloud_xLWRIST = cloud_pose_estimation['frames'][0]['persons'][0]['pose2d']['joints'][30]\n",
    "cloud_xLKNEE = cloud_pose_estimation['frames'][0]['persons'][0]['pose2d']['joints'][8]\n",
    "\n",
    "print(cloud_xLWRIST)\n",
    "print(cloud_xLKNEE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "-> Now you can use human pose estimation in your project with very few line of code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Getting started with the edge sdk\n",
    "\n",
    "The Edge SDK allows you to do real-time pose estimation.\n",
    "\n",
    "For installation procedure refer to the detailed documentation here: \n",
    "https://devportal.wrnch.ai/wrnchai_sdk/releases\n",
    "\n",
    "NB: We do not support MacOS yet.\n",
    "\n",
    "Common pitfalls when working with the Python SDK\n",
    "- `No module called wrnchAI` : Ensure that lib/python/X.X is in your PYTHONPATH. Or simply, on the top of your python script do\n",
    "  ```python\n",
    "  import sys\n",
    "  sys.path.append('<path_to_project_root>/lib/python/X.X')\n",
    "  ```\n",
    "- License problems : Ensure the license is under your home directory or set as an environment variable\n",
    "- When working with python, it's good practice to create a virtual environment.\n",
    "\n",
    "For example with Python3:\n",
    "\n",
    "   - `python -m venv <env_name>`\n",
    "\n",
    "   - Then activate it by running the activate script in the bin directory\n",
    "    \n",
    "   - Then add opencv dependency for the samples with this command: `pip install opencv-python`\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}