examples/Demo_labeledexample_Openfield.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "N0gDJMy1ywm8"
   },
   "source": [
    "# DeepLabCut Toolbox - Open-Field DEMO\n",
    "https://github.com/AlexEMG/DeepLabCut\n",
    "\n",
    "#### The notebook accompanies the following user-guide:\n",
    "\n",
    "Nath\\*, Mathis\\* et al. *Using DeepLabCut for markerless pose estimation during behavior across species* Nature Protocols, 2019: https://www.nature.com/articles/s41596-019-0176-0\n",
    "\n",
    "This notebook illustrates how to:\n",
    "- load the demo project\n",
    "- train a network\n",
    "- evaluate a network\n",
    "- analyze a novel video\n",
    "- create an automatically labeled video \n",
    "- plot the trajectories \n",
    "- identify outlier frames\n",
    "- annotate the outlier frames manually\n",
    "- merge the data sets and update the training set\n",
    "- train a network\n",
    "\n",
    "Note: This notebook starts from an already initialized project with labeled data.\n",
    "\n",
    "\n",
    "The data is a subset from *DeepLabCut: markerless pose estimation of user-defined body parts with deep learning* https://www.nature.com/articles/s41593-018-0209-y (this subset was not used to train models that are shown or evaluated in our paper)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hTtJxcQ7ywnB"
   },
   "outputs": [],
   "source": [
    "# Importing the toolbox (takes several seconds)\n",
    "import deeplabcut"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WOEHc0MeywnJ"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded, now creating training data...\n",
      "/home/mackenzie/DEEPLABCUT/3D/DeepLabCut2.0-master/examples/openfield-Pranav-2018-10-30/training-datasets/iteration-0/UnaugmentedDataSet_openfieldOct30  already exists!\n",
      "/home/mackenzie/DEEPLABCUT/3D/DeepLabCut2.0-master/examples/openfield-Pranav-2018-10-30/labeled-data/short_mp3/CollectedData_Pranav.h5  not found (perhaps not annotated)\n",
      "/home/mackenzie/DEEPLABCUT/3D/DeepLabCut2.0-master/examples/openfield-Pranav-2018-10-30/dlc-models/iteration-0/openfieldOct30-trainset95shuffle1  already exists!\n",
      "/home/mackenzie/DEEPLABCUT/3D/DeepLabCut2.0-master/examples/openfield-Pranav-2018-10-30/dlc-models/iteration-0/openfieldOct30-trainset95shuffle1//train  already exists!\n",
      "/home/mackenzie/DEEPLABCUT/3D/DeepLabCut2.0-master/examples/openfield-Pranav-2018-10-30/dlc-models/iteration-0/openfieldOct30-trainset95shuffle1//test  already exists!\n",
      "The training dataset is successfully created. Use the function 'train_network' to start training. Happy training!\n"
     ]
    }
   ],
   "source": [
    "# Loading example data set:\n",
    "import os\n",
    "# Note that parameters of this project can be seen at: *openfield-Pranav-2018-10-30/config.yaml*\n",
    "from pathlib import Path\n",
    "path_config_file = os.path.join(os.getcwd(),'openfield-Pranav-2018-10-30/config.yaml')\n",
    "deeplabcut.load_demo_data(path_config_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ROlflqQLywnP"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating images with labels by Pranav.\n",
      "/home/mackenzie/DEEPLABCUT/3D/DeepLabCut2.0-master/examples/openfield-Pranav-2018-10-30/labeled-data/m4s1_labeled  already exists!\n",
      "They are stored in the following folder: /home/mackenzie/DEEPLABCUT/3D/DeepLabCut2.0-master/examples/openfield-Pranav-2018-10-30/labeled-data/m4s1_labeled.\n",
      "Attention: /home/mackenzie/DEEPLABCUT/3D/DeepLabCut2.0-master/examples/openfield-Pranav-2018-10-30/labeled-data/short_mp3 does not appear to have labeled data!\n",
      "If all the labels are ok, then use the function 'create_training_dataset' to create the training dataset!\n"
     ]
    }
   ],
   "source": [
    "#[OPTIONAL] Perhaps plot the labels to see how the frames were annotated:\n",
    "#(note, this project was created in Linux, so you might have an error in Windows, but this is an optional step)\n",
    "deeplabcut.check_labels(path_config_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "h9H7eqDLywnV"
   },
   "source": [
    "## Start training of Feature Detectors\n",
    "This function trains the network for a specific shuffle of the training dataset. The user can set various parameters in */openfield-Pranav-2018-10-30/dlc-models/.../pose_cfg.yaml*. \n",
    "\n",
    "Training can be stopped at any time. Note that the weights are only stored every 'save_iters' steps. For this demo the state it is advisable to store & display the progress very often. In practice this is inefficient.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jg96O2acywnW",
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Config:\n",
      "{'all_joints': [[0], [1], [2], [3]],\n",
      " 'all_joints_names': ['snout', 'leftear', 'rightear', 'tailbase'],\n",
      " 'batch_size': 1,\n",
      " 'bottomheight': 400,\n",
      " 'crop': True,\n",
      " 'crop_pad': 0,\n",
      " 'cropratio': 0.4,\n",
      " 'dataset': 'training-datasets/iteration-0/UnaugmentedDataSet_openfieldOct30/openfield_Pranav95shuffle1.mat',\n",
      " 'dataset_type': 'default',\n",
      " 'display_iters': 1000,\n",
      " 'fg_fraction': 0.25,\n",
      " 'global_scale': 0.8,\n",
      " 'init_weights': '/home/mackenzie/anaconda3/envs/DLC2/lib/python3.6/site-packages/deeplabcut/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt',\n",
      " 'intermediate_supervision': False,\n",
      " 'intermediate_supervision_layer': 12,\n",
      " 'leftwidth': 400,\n",
      " 'location_refinement': True,\n",
      " 'locref_huber_loss': True,\n",
      " 'locref_loss_weight': 0.05,\n",
      " 'locref_stdev': 7.2801,\n",
      " 'log_dir': 'log',\n",
      " 'max_input_size': 1500,\n",
      " 'mean_pixel': [123.68, 116.779, 103.939],\n",
      " 'metadataset': 'training-datasets/iteration-0/UnaugmentedDataSet_openfieldOct30/Documentation_data-openfield_95shuffle1.pickle',\n",
      " 'min_input_size': 64,\n",
      " 'minsize': 100,\n",
      " 'mirror': False,\n",
      " 'multi_step': [[0.005, 10000],\n",
      "                [0.02, 430000],\n",
      "                [0.002, 730000],\n",
      "                [0.001, 1030000]],\n",
      " 'net_type': 'resnet_50',\n",
      " 'num_joints': 4,\n",
      " 'optimizer': 'sgd',\n",
      " 'pos_dist_thresh': 17,\n",
      " 'project_path': '/home/mackenzie/DEEPLABCUT/3D/DeepLabCut2.0-master/examples/openfield-Pranav-2018-10-30',\n",
      " 'regularize': False,\n",
      " 'rightwidth': 400,\n",
      " 'save_iters': 50000,\n",
      " 'scale_jitter_lo': 0.5,\n",
      " 'scale_jitter_up': 1.25,\n",
      " 'scoremap_dir': 'test',\n",
      " 'shuffle': True,\n",
      " 'snapshot_prefix': '/home/mackenzie/DEEPLABCUT/3D/DeepLabCut2.0-master/examples/openfield-Pranav-2018-10-30/dlc-models/iteration-0/openfieldOct30-trainset95shuffle1/train/snapshot',\n",
      " 'stride': 8.0,\n",
      " 'topheight': 400,\n",
      " 'use_gt_segm': False,\n",
      " 'video': False,\n",
      " 'video_batch': False,\n",
      " 'weigh_negatives': False,\n",
      " 'weigh_only_present_joints': False,\n",
      " 'weigh_part_predictions': False,\n",
      " 'weight_decay': 0.0001}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from /home/mackenzie/anaconda3/envs/DLC2/lib/python3.6/site-packages/deeplabcut/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt\n",
      "Display_iters overwritten as 10\n",
      "Save_iters overwritten as 100\n",
      "Training parameter:\n",
      "{'stride': 8.0, 'weigh_part_predictions': False, 'weigh_negatives': False, 'fg_fraction': 0.25, 'weigh_only_present_joints': False, 'mean_pixel': [123.68, 116.779, 103.939], 'shuffle': True, 'snapshot_prefix': '/home/mackenzie/DEEPLABCUT/3D/DeepLabCut2.0-master/examples/openfield-Pranav-2018-10-30/dlc-models/iteration-0/openfieldOct30-trainset95shuffle1/train/snapshot', 'log_dir': 'log', 'global_scale': 0.8, 'location_refinement': True, 'locref_stdev': 7.2801, 'locref_loss_weight': 0.05, 'locref_huber_loss': True, 'optimizer': 'sgd', 'intermediate_supervision': False, 'intermediate_supervision_layer': 12, 'regularize': False, 'weight_decay': 0.0001, 'mirror': False, 'crop_pad': 0, 'scoremap_dir': 'test', 'dataset_type': 'default', 'use_gt_segm': False, 'batch_size': 1, 'video': False, 'video_batch': False, 'crop': True, 'cropratio': 0.4, 'minsize': 100, 'leftwidth': 400, 'rightwidth': 400, 'topheight': 400, 'bottomheight': 400, 'all_joints': [[0], [1], [2], [3]], 'all_joints_names': ['snout', 'leftear', 'rightear', 'tailbase'], 'dataset': 'training-datasets/iteration-0/UnaugmentedDataSet_openfieldOct30/openfield_Pranav95shuffle1.mat', 'display_iters': 1000, 'init_weights': '/home/mackenzie/anaconda3/envs/DLC2/lib/python3.6/site-packages/deeplabcut/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt', 'max_input_size': 1500, 'metadataset': 'training-datasets/iteration-0/UnaugmentedDataSet_openfieldOct30/Documentation_data-openfield_95shuffle1.pickle', 'min_input_size': 64, 'multi_step': [[0.005, 10000], [0.02, 430000], [0.002, 730000], [0.001, 1030000]], 'net_type': 'resnet_50', 'num_joints': 4, 'pos_dist_thresh': 17, 'project_path': '/home/mackenzie/DEEPLABCUT/3D/DeepLabCut2.0-master/examples/openfield-Pranav-2018-10-30', 'save_iters': 50000, 'scale_jitter_lo': 0.5, 'scale_jitter_up': 1.25}\n",
      "Starting training....\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "iteration: 10 loss: 0.3308 lr: 0.005\n",
      "iteration: 20 loss: 0.0563 lr: 0.005\n",
      "iteration: 30 loss: 0.0417 lr: 0.005\n",
      "iteration: 40 loss: 0.0362 lr: 0.005\n",
      "iteration: 50 loss: 0.0407 lr: 0.005\n",
      "iteration: 60 loss: 0.0461 lr: 0.005\n",
      "iteration: 70 loss: 0.0385 lr: 0.005\n",
      "iteration: 80 loss: 0.0345 lr: 0.005\n",
      "iteration: 90 loss: 0.0314 lr: 0.005\n",
      "iteration: 100 loss: 0.0428 lr: 0.005\n",
      "iteration: 110 loss: 0.0262 lr: 0.005\n",
      "iteration: 120 loss: 0.0255 lr: 0.005\n",
      "iteration: 130 loss: 0.0275 lr: 0.005\n",
      "iteration: 140 loss: 0.0251 lr: 0.005\n",
      "iteration: 150 loss: 0.0221 lr: 0.005\n",
      "iteration: 160 loss: 0.0209 lr: 0.005\n",
      "iteration: 170 loss: 0.0297 lr: 0.005\n",
      "iteration: 180 loss: 0.0325 lr: 0.005\n",
      "iteration: 190 loss: 0.0242 lr: 0.005\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-4-e730f91a267a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mdeeplabcut\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain_network\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath_config_file\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mshuffle\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdisplayiters\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msaveiters\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/home/mackenzie/anaconda3/envs/DLC2/lib/python3.6/site-packages/deeplabcut/pose_estimation_tensorflow/training.py\u001b[0m in \u001b[0;36mtrain_network\u001b[0;34m(config, shuffle, trainingsetindex, gputouse, max_snapshots_to_keep, autotune, displayiters, saveiters, maxiters)\u001b[0m\n\u001b[1;32m     87\u001b[0m           \u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mposeconfigfile\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mdisplayiters\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0msaveiters\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mmaxiters\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mmax_to_keep\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmax_snapshots_to_keep\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m#pass on path and file name for pose_cfg.yaml!\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     88\u001b[0m       \u001b[0;32mexcept\u001b[0m \u001b[0mBaseException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 89\u001b[0;31m           \u001b[0;32mraise\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     90\u001b[0m       \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     91\u001b[0m           \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchdir\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstart_path\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mackenzie/anaconda3/envs/DLC2/lib/python3.6/site-packages/deeplabcut/pose_estimation_tensorflow/training.py\u001b[0m in \u001b[0;36mtrain_network\u001b[0;34m(config, shuffle, trainingsetindex, gputouse, max_snapshots_to_keep, autotune, displayiters, saveiters, maxiters)\u001b[0m\n\u001b[1;32m     85\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     86\u001b[0m       \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 87\u001b[0;31m           \u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mposeconfigfile\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mdisplayiters\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0msaveiters\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mmaxiters\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mmax_to_keep\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmax_snapshots_to_keep\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m#pass on path and file name for pose_cfg.yaml!\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     88\u001b[0m       \u001b[0;32mexcept\u001b[0m \u001b[0mBaseException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     89\u001b[0m           \u001b[0;32mraise\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mackenzie/anaconda3/envs/DLC2/lib/python3.6/site-packages/deeplabcut/pose_estimation_tensorflow/train.py\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(config_yaml, displayiters, saveiters, maxiters, max_to_keep)\u001b[0m\n\u001b[1;32m    140\u001b[0m         \u001b[0mcurrent_lr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlr_gen\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_lr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mit\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    141\u001b[0m         [_, loss_val, summary] = sess.run([train_op, total_loss, merged_summaries],\n\u001b[0;32m--> 142\u001b[0;31m                                           feed_dict={learning_rate: current_lr})\n\u001b[0m\u001b[1;32m    143\u001b[0m         \u001b[0mcum_loss\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mloss_val\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    144\u001b[0m         \u001b[0mtrain_writer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_summary\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msummary\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mit\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mackenzie/anaconda3/envs/DLC2/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    898\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    899\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 900\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    901\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    902\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mackenzie/anaconda3/envs/DLC2/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1133\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mfeed_dict_tensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1134\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m-> 1135\u001b[0;31m                              feed_dict_tensor, options, run_metadata)\n\u001b[0m\u001b[1;32m   1136\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1137\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mackenzie/anaconda3/envs/DLC2/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1314\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1315\u001b[0m       return self._do_call(_run_fn, feeds, fetches, targets, options,\n\u001b[0;32m-> 1316\u001b[0;31m                            run_metadata)\n\u001b[0m\u001b[1;32m   1317\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1318\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_prun_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeeds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetches\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mackenzie/anaconda3/envs/DLC2/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1320\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1321\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1322\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1323\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1324\u001b[0m       \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mackenzie/anaconda3/envs/DLC2/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1305\u001b[0m       \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_extend_graph\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1306\u001b[0m       return self._call_tf_sessionrun(\n\u001b[0;32m-> 1307\u001b[0;31m           options, feed_dict, fetch_list, target_list, run_metadata)\n\u001b[0m\u001b[1;32m   1308\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1309\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mackenzie/anaconda3/envs/DLC2/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_call_tf_sessionrun\u001b[0;34m(self, options, feed_dict, fetch_list, target_list, run_metadata)\u001b[0m\n\u001b[1;32m   1407\u001b[0m       return tf_session.TF_SessionRun_wrapper(\n\u001b[1;32m   1408\u001b[0m           \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptions\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1409\u001b[0;31m           run_metadata)\n\u001b[0m\u001b[1;32m   1410\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1411\u001b[0m       \u001b[0;32mwith\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraise_exception_on_not_ok_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mstatus\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "deeplabcut.train_network(path_config_file, shuffle=1, displayiters=10, saveiters=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note, that if it reaches the end or you stop it (by hitting \"stop\" or by CTRL+C), \n",
    "you will see an \"KeyboardInterrupt\" error, but you can ignore this!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CCzxVT_gywnc"
   },
   "source": [
    "## Evaluate a trained network\n",
    "\n",
    "This function evaluates a trained model for a specific shuffle/shuffles at a particular training state (snapshot) or on all the states. The network is evaluated on the data set (images) and stores the results as .csv file in a subdirectory under **evaluation-results**.\n",
    "\n",
    "You can change various parameters in the ```config.yaml``` file of this project. For evaluation all the model descriptors (Task, TrainingFraction, Date etc.) are important. For the evaluation one can change pcutoff. This cutoff also influences how likely estimated postions need to be so that they are shown in the plots. One can furthermore, change the colormap and dotsize for those graphs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kuprPKDdywne",
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Config:\n",
      "{'all_joints': [[0], [1], [2], [3]],\n",
      " 'all_joints_names': ['snout', 'leftear', 'rightear', 'tailbase'],\n",
      " 'batch_size': 1,\n",
      " 'bottomheight': 400,\n",
      " 'crop': True,\n",
      " 'crop_pad': 0,\n",
      " 'cropratio': 0.4,\n",
      " 'dataset': 'training-datasets/iteration-0/UnaugmentedDataSet_openfieldOct30/openfield_Pranav95shuffle1.mat',\n",
      " 'dataset_type': 'default',\n",
      " 'display_iters': 1000,\n",
      " 'fg_fraction': 0.25,\n",
      " 'global_scale': 0.8,\n",
      " 'init_weights': '/home/mackenzie/anaconda3/envs/DLC2/lib/python3.6/site-packages/deeplabcut/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt',\n",
      " 'intermediate_supervision': False,\n",
      " 'intermediate_supervision_layer': 12,\n",
      " 'leftwidth': 400,\n",
      " 'location_refinement': True,\n",
      " 'locref_huber_loss': True,\n",
      " 'locref_loss_weight': 0.05,\n",
      " 'locref_stdev': 7.2801,\n",
      " 'log_dir': 'log',\n",
      " 'max_input_size': 1500,\n",
      " 'mean_pixel': [123.68, 116.779, 103.939],\n",
      " 'metadataset': 'training-datasets/iteration-0/UnaugmentedDataSet_openfieldOct30/Documentation_data-openfield_95shuffle1.pickle',\n",
      " 'min_input_size': 64,\n",
      " 'minsize': 100,\n",
      " 'mirror': False,\n",
      " 'multi_step': [[0.005, 10000],\n",
      "                [0.02, 430000],\n",
      "                [0.002, 730000],\n",
      "                [0.001, 1030000]],\n",
      " 'net_type': 'resnet_50',\n",
      " 'num_joints': 4,\n",
      " 'optimizer': 'sgd',\n",
      " 'pos_dist_thresh': 17,\n",
      " 'project_path': '/home/mackenzie/DEEPLABCUT/3D/DeepLabCut2.0-master/examples/openfield-Pranav-2018-10-30',\n",
      " 'regularize': False,\n",
      " 'rightwidth': 400,\n",
      " 'save_iters': 50000,\n",
      " 'scale_jitter_lo': 0.5,\n",
      " 'scale_jitter_up': 1.25,\n",
      " 'scoremap_dir': 'test',\n",
      " 'shuffle': True,\n",
      " 'snapshot_prefix': '/home/mackenzie/DEEPLABCUT/3D/DeepLabCut2.0-master/examples/openfield-Pranav-2018-10-30/dlc-models/iteration-0/openfieldOct30-trainset95shuffle1/test/snapshot',\n",
      " 'stride': 8.0,\n",
      " 'topheight': 400,\n",
      " 'use_gt_segm': False,\n",
      " 'video': False,\n",
      " 'video_batch': False,\n",
      " 'weigh_negatives': False,\n",
      " 'weigh_only_present_joints': False,\n",
      " 'weigh_part_predictions': False,\n",
      " 'weight_decay': 0.0001}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/mackenzie/DEEPLABCUT/3D/DeepLabCut2.0-master/examples/openfield-Pranav-2018-10-30/evaluation-results/  already exists!\n",
      "/home/mackenzie/DEEPLABCUT/3D/DeepLabCut2.0-master/examples/openfield-Pranav-2018-10-30/evaluation-results/iteration-0/openfieldOct30-trainset95shuffle1  already exists!\n",
      "Running  DeepCut_resnet50_openfieldOct30shuffle1_2400  with # of trainingiterations: 2400\n",
      "This net has already been evaluated!\n",
      "The network is evaluated and the results are stored in the subdirectory 'evaluation_results'.\n",
      "If it generalizes well, choose the best model for prediction and update the config file with the appropriate index for the 'snapshotindex'.\n",
      "Use the function 'analyze_video' to make predictions on new videos.\n",
      "Otherwise consider retraining the network (see DeepLabCut workflow Fig 2)\n"
     ]
    }
   ],
   "source": [
    "deeplabcut.evaluate_network(path_config_file,plotting=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*NOTE: depending on your set up sometimes you get some \"matplotlib errors, but these are not important*\n",
    "\n",
    "Now you can go check out the images. Given the limted data input and it took ~20 mins to test this out, it is not meant to track well, so don't be alarmed. This is just to get you familiar with the workflow... "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XeqYWGaXywnj"
   },
   "source": [
    "## Analyzing videos\n",
    "This function extracts the pose based on a trained network from videos. The user can choose the trained network - by default the most recent snapshot is used to analyse the videos. However, the user can also specify the snapshot index for the variable **snapshotindex** in the **config.yaml** file).\n",
    "\n",
    "The results are stored in hd5 file in the same directory, where the video resides. The pose array (pose vs. frame index) can also be exported as csv file (set flag to...). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Vv9iHHLlywnl"
   },
   "outputs": [],
   "source": [
    "# Creating video path:\n",
    "import os\n",
    "videofile_path = os.path.join(os.getcwd(),'openfield-Pranav-2018-10-30/videos/m3v1mp4.mp4')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CFbPPD4hywnq",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "print(\"Start analyzing the video!\")\n",
    "#our demo video on a CPU with take ~30 min to analze! GPU is much faster!\n",
    "deeplabcut.analyze_videos(path_config_file,[videofile_path])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QQ3T3oykywnw"
   },
   "source": [
    "## Create labeled video\n",
    "\n",
    "This function is for the visualization purpose and can be used to create a video in .mp4 format with the predicted labels. This video is saved in the same directory, where the (unlabeled) video resides. \n",
    "\n",
    "Various parameters can be set with regard to the colormap and the dotsize. The parameters of the "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GhI9KLs4ywn0",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "deeplabcut.create_labeled_video(path_config_file,[videofile_path])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IErvm1K5ywn5"
   },
   "source": [
    "## Plot the trajectories of the analyzed videos\n",
    "This function plots the trajectories of all the body parts across the entire video. Each body part is identified by a unique color. The underlying functions can easily be customized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mP2useJgywn7",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%matplotlib notebook\n",
    "deeplabcut.plot_trajectories(path_config_file,[videofile_path],showfigures=True)\n",
    "\n",
    "#These plots can are interactive and can be customized (see https://matplotlib.org/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "a5nOaWzXywoB"
   },
   "source": [
    "## Extract outlier frames, where the predictions are off.\n",
    "\n",
    "This is optional step allows to add more training data when the evaluation results are poor. In such a case, the user can use the following function to extract frames where the labels are incorrectly predicted. Make sure to provide the correct value of the \"iterations\" as it will be used to create the unique directory where the extracted frames will be saved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RJGiDKuUywoC",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "deeplabcut.extract_outlier_frames(path_config_file,[videofile_path])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jHjpscPcywoG"
   },
   "source": [
    "The user can run this iteratively, and (even) extract additional frames from the same video."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uaNUm3NSywoH"
   },
   "source": [
    "## Manually correct labels\n",
    "\n",
    "This step allows the user to correct the labels in the extracted frames. Navigate to the folder corresponding to the video 'MovieS2_Perturbation_noLaser_compressed' and use the GUI as described in the protocol to update the labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OJDvJMcrywoI"
   },
   "outputs": [],
   "source": [
    "%gui wx\n",
    "deeplabcut.refine_labels(path_config_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Y7efellnywoT"
   },
   "outputs": [],
   "source": [
    "#Perhaps plot the labels to see how how all the frames are annoted (including the refined ones)\n",
    "deeplabcut.check_labels(path_config_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mcuqoeRbywoL"
   },
   "outputs": [],
   "source": [
    "# Now merge datasets (once you refined all frames)\n",
    "deeplabcut.merge_datasets(path_config_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WRB9KgGsywoP"
   },
   "source": [
    "## Create a new iteration of training dataset, check it and train...\n",
    "\n",
    "Following the refine labels, append these frames to the original dataset to create a new iteration of training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NGHghXfdywoQ"
   },
   "outputs": [],
   "source": [
    "deeplabcut.create_training_dataset(path_config_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8fhL6nG2ywoW"
   },
   "source": [
    "Now one can train the network again... (with the expanded data set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cAUxg5sgywoX"
   },
   "outputs": [],
   "source": [
    "deeplabcut.train_network(path_config_file, shuffle=1)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "Demo-labeledexample-MouseReaching.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python [conda env:DLC2]",
   "language": "python",
   "name": "conda-env-DLC2-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}