sfm_analysis_final.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# CSE252C: Homework 1\n",
    "## Computing Resources\n",
    "Please read the README file of this repository for the instructions\n",
    "## Instructions\n",
    "1. Attempt all questions.\n",
    "2. Please comment all your code adequately.\n",
    "3. Include all relevant information such as text answers, output images in notebook.\n",
    "4. **Academic integrity:** The homework must be completed individually.\n",
    "\n",
    "5. **Submission instructions:**  \n",
    " (a) Submit the notebook and its PDF version on Gradescope.  \n",
    " (b) Rename your submission files as Lastname_Firstname.ipynb and Lastname_Firstname.pdf.  \n",
    " (c) Correctly select pages for each answer on Gradescope to allow proper grading.\n",
    "\n",
    "6. **Due date:** Assignments are due Mon, May 4, by 4pm PST."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Question 1: Warm Up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "We will first try SFM using the original implementation from $\\mathtt{libviso2}$[8],[9]. We will test on a dataset containing 300 images from one sequence of the KITTI dataset with ground-truth camera poses and camera calibration information. \n",
    "\n",
    "Run the SFM algorithm using the following script. You are required to report two error metrics. The error metric for rotation is defined as the mean of Frobenius norm of the difference between the ground-truth rotation matrix and predicted rotation matrix. The error metric for translation is defined as mean of the L2 distance. Both errors will be printed on the screen as you run the code.  **(5 points)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/sgaba/sfm/pyviso\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import sys\n",
    "# change your base path\n",
    "os.chdir('./pyviso/') # './'\n",
    "print(os.getcwd())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "np.set_printoptions(precision=4)\n",
    "np.set_printoptions(suppress=True)\n",
    "import viso2\n",
    "import matplotlib.pyplot as plt\n",
    "from skimage.io import imread\n",
    "import time\n",
    "\n",
    "def errorMetric(RPred, RGt, TPred, TGt):\n",
    "    diffRot = (RPred - RGt)\n",
    "    diffTrans = (TPred - TGt)\n",
    "    errorRot = np.sqrt(np.sum(np.multiply(diffRot.reshape(-1), diffRot.reshape(-1))))\n",
    "    errorTrans = np.sqrt(np.sum(np.multiply(diffTrans.reshape(-1), diffTrans.reshape(-1))))\n",
    "\n",
    "    return errorRot, errorTrans\n",
    "\n",
    "if_vis = False # set to True to do the visualization per frame; the images will be saved at '.vis/'. Turn off if you just want the camera poses and errors\n",
    "if_on_screen = False # if True the visualization per frame is going to be displayed realtime on screen; if False there will be no display, but in both options the images will be saved\n",
    "\n",
    "# parameter settings (for an example, please download\n",
    "# dataset_path = '../dataset_SfM/'\n",
    "dataset_path = '/datasets/cse152-252-sp20-public/dataset_SfM' # On the ``ieng6.ucsd.edu`` server\n",
    "img_dir      = os.path.join(dataset_path, 'sequences/00/image_0')\n",
    "gt_dir       = os.path.join(dataset_path, 'poses/00.txt')\n",
    "calibFile    = os.path.join(dataset_path, 'sequences/00/calib.txt')\n",
    "border       = 50;\n",
    "gap          = 15;\n",
    "\n",
    "# Load the camera calibration information\n",
    "with open(calibFile) as fid:\n",
    "    calibLines = fid.readlines()\n",
    "    calibLines = [calibLine.strip() for calibLine in calibLines]\n",
    "\n",
    "calibInfo = [float(calibStr) for calibStr in calibLines[0].split(' ')[1:]]\n",
    "# param = {'f': calibInfo[0], 'cu': calibInfo[2], 'cv': calibInfo[6]}\n",
    "\n",
    "# Load the ground-truth depth and rotation\n",
    "with open(gt_dir) as fid:\n",
    "    gtTr = [[float(TrStr) for TrStr in line.strip().split(' ')] for line in fid.readlines()]\n",
    "gtTr = np.asarray(gtTr).reshape(-1, 3, 4)\n",
    "\n",
    "# param['height'] = 1.6\n",
    "# param['pitch']  = -0.08\n",
    "# param['match'] = {'pre_step_size': 64}\n",
    "first_frame  = 0\n",
    "last_frame   = 300\n",
    "epi = 1e-8\n",
    "\n",
    "# init visual odometry\n",
    "params = viso2.Mono_parameters()\n",
    "params.calib.f = calibInfo[0]\n",
    "params.calib.cu = calibInfo[2]\n",
    "params.calib.cv = calibInfo[6]\n",
    "params.height = 1.6\n",
    "params.pitch = -0.08\n",
    "\n",
    "\n",
    "first_frame  = 0\n",
    "last_frame   = 300\n",
    "\n",
    "# init transformation matrix array\n",
    "Tr_total = []\n",
    "Tr_total_np = []\n",
    "Tr_total.append(viso2.Matrix_eye(4))\n",
    "Tr_total_np.append(np.eye(4))\n",
    "\n",
    "# init viso module\n",
    "visoMono = viso2.VisualOdometryMono(params)\n",
    "\n",
    "if if_vis:\n",
    "    save_path = 'vis'\n",
    "    os.makedirs(save_path, exist_ok=True)\n",
    "\n",
    "    # create figure\n",
    "    fig = plt.figure(figsize=(10, 15))\n",
    "    ax1 = plt.subplot(211)\n",
    "    ax1.axis('off')\n",
    "    ax2 = plt.subplot(212)\n",
    "    ax2.set_xticks(np.arange(-100, 100, step=10))\n",
    "    ax2.set_yticks(np.arange(-500, 500, step=10))\n",
    "    ax2.axis('equal')\n",
    "    ax2.grid()\n",
    "    if if_on_screen:\n",
    "        plt.ion()\n",
    "    else:\n",
    "        plt.ioff()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# for all frames do\n",
    "if_replace = False\n",
    "errorTransSum = 0\n",
    "errorRotSum = 0\n",
    "errorRot_list = []\n",
    "errorTrans_list =[]\n",
    "\n",
    "#initialize an empty df to report errors\n",
    "errors_df = pd.DataFrame(columns = [\"Mean Rotation Error\",\"Mean Transition Error\"])\n",
    "\n",
    "for frame in range(first_frame, last_frame):\n",
    "#     break\n",
    "    # 1-based index\n",
    "    k = frame-first_frame+1\n",
    "\n",
    "    # read current images\n",
    "    I = imread(os.path.join(img_dir, '%06d.png'%frame))\n",
    "    assert(len(I.shape) == 2) # should be grayscale\n",
    "\n",
    "    # compute egomotion\n",
    "    process_result = visoMono.process_frame(I, if_replace)\n",
    "    Tr = visoMono.getMotion()\n",
    "    matrixer = viso2.Matrix(Tr)\n",
    "    Tr_np = np.zeros((4, 4))\n",
    "    Tr.toNumpy(Tr_np) # so awkward...\n",
    "\n",
    "    # accumulate egomotion, starting with second frame\n",
    "    if k > 1:\n",
    "        if process_result is False:\n",
    "            if_replace = True\n",
    "            Tr_total.append(Tr_total[-1])\n",
    "            Tr_total_np.append(Tr_total_np[-1])\n",
    "        else:\n",
    "            if_replace = False\n",
    "            Tr_total.append(Tr_total[-1] * viso2.Matrix_inv(Tr))\n",
    "            Tr_total_np.append(Tr_total_np[-1] @ np.linalg.inv(Tr_np)) # should be the same\n",
    "            print(Tr_total_np[-1])\n",
    "\n",
    "    # output statistics\n",
    "    num_matches = visoMono.getNumberOfMatches()\n",
    "    num_inliers = visoMono.getNumberOfInliers()\n",
    "    matches = visoMono.getMatches()\n",
    "    matches_np = np.empty([4, matches.size()])\n",
    "\n",
    "    for i,m in enumerate(matches):\n",
    "        matches_np[:, i] = (m.u1p, m.v1p, m.u1c, m.v1c)\n",
    "\n",
    "    if if_vis:\n",
    "        # update image\n",
    "        ax1.clear()\n",
    "        ax1.imshow(I, cmap='gray', vmin=0, vmax=255)\n",
    "        if num_matches != 0:\n",
    "            for n in range(num_matches):\n",
    "                ax1.plot([matches_np[0, n], matches_np[2, n]], [matches_np[1, n], matches_np[3, n]])\n",
    "        ax1.set_title('Frame %d'%frame)\n",
    "\n",
    "        # update trajectory\n",
    "        if k > 1:\n",
    "            ax2.plot([Tr_total_np[k-2][0, 3], Tr_total_np[k-1][0, 3]], \\\n",
    "                [Tr_total_np[k-2][2, 3], Tr_total_np[k-1][2, 3]], 'b.-', linewidth=1)\n",
    "            ax2.plot([gtTr[k-2][0, 3], gtTr[k-1][0, 3]], \\\n",
    "                [gtTr[k-2][2, 3], gtTr[k-1][2, 3]], 'r.-', linewidth=1)\n",
    "        ax2.set_title('Blue: estimated trajectory; Red: ground truth trejectory')\n",
    "\n",
    "        plt.draw()\n",
    "\n",
    "    # Compute rotation\n",
    "    Rpred_p = Tr_total_np[k-2][0:3, 0:3]\n",
    "    Rpred_c = Tr_total_np[k-1][0:3, 0:3]\n",
    "    Rpred = Rpred_c.transpose() @ Rpred_p\n",
    "    Rgt_p = np.squeeze(gtTr[k-2, 0:3, 0:3])\n",
    "    Rgt_c = np.squeeze(gtTr[k-1, 0:3, 0:3])\n",
    "    Rgt = Rgt_c.transpose() @ Rgt_p\n",
    "    # Compute translation\n",
    "    Tpred_p = Tr_total_np[k-2][0:3, 3:4]\n",
    "    Tpred_c = Tr_total_np[k-1][0:3, 3:4]\n",
    "    Tpred = Tpred_c - Tpred_p\n",
    "    Tgt_p = gtTr[k-2, 0:3, 3:4]\n",
    "    Tgt_c = gtTr[k-1, 0:3, 3:4]\n",
    "    Tgt = Tgt_c - Tgt_p\n",
    "    # Compute errors\n",
    "    errorRot, errorTrans = errorMetric(Rpred, Rgt, Tpred, Tgt)\n",
    "    errorRotSum = errorRotSum + errorRot\n",
    "    errorTransSum = errorTransSum + errorTrans\n",
    "    # errorRot_list.append(errorRot)\n",
    "    # errorTrans_list.append(errorTrans)\n",
    "    meanRotError = errorRotSum / (k-1+epi)\n",
    "    meanTransError = errorTransSum / (k-1+epi)\n",
    "    print('Mean Error Rotation: %.5f'%(meanRotError))\n",
    "    print('Mean Error Translation: %.5f'%(meanTransError))\n",
    "\n",
    "    errors_df.loc[len(errors_df)] = [meanRotError,meanTransError]\n",
    "\n",
    "    print('== [Result] Frame: %d, Matches %d, Inliers: %.2f'%(frame, num_matches, 100*num_inliers/(num_matches+1e-8)))\n",
    "\n",
    "    if if_vis:\n",
    "        # input('Paused; Press Enter to continue') # Option 1: Manually pause and resume\n",
    "        if if_on_screen:\n",
    "            plt.pause(0.1) # Or Option 2: enable to this to auto pause for a while after daring to enable animation in case of a delay in drawing\n",
    "        vis_path = os.path.join(save_path, 'frame%03d.jpg'%frame)\n",
    "        fig.savefig(vis_path)\n",
    "        print('Saved at %s'%vis_path)\n",
    "        \n",
    "        if frame % 50 == 0 or frame == last_frame-1:\n",
    "            plt.figure(figsize=(10, 15))\n",
    "            plt.imshow(plt.imread(vis_path))\n",
    "            plt.axis('off')\n",
    "            plt.show()\n",
    "\n",
    "\n",
    "# input('Press Enter to exit')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "1. Report the final rotation and translation error. **(2 points)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "* Final Frame Mean Rotation and Mean Translation Error are 0.00277 and 0.53721 respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Not to be reported'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"Not to be reported\"\"\"\n",
    "# print(\"Rotation and Transition Error metric for all frames \\n\")\n",
    "# errors_df.index = np.arange(1, len(errors_df) + 1)\n",
    "# errors_df.index.name = \"Frame Number\"\n",
    "# with pd.option_context('display.max_rows', None, 'display.max_columns', None):\n",
    "#     print(errors_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Then answer the questions below"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "2. In $\\mathtt{libviso2}$, the feature points are \"bucketed\" ($\\mathtt{src/matcher.cpp: Line 285 - 326}$), which means in a certain area of region, the number of detected keypoint pairs should be within certain bounds. Why?  **(3 points)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "* In bucketing, the image is divided into several nonoverlapping rectangles and in every bucket we keep a maximal number of feature points. First, the smaller number of features reduces the computational complexity of the algorithm which is an important prerequisite for real time applications. Second, this technique guarantees that the used image features are well distributed along the z-axis, i.e. the roll-axis of the vehicle. This turns out to be important for a good estimation of the linear and angluar velocities. The distribution of image features along the z-axis ensures that far as well as near features are used for the estimation process. This results in a precise estimation of the overall egomotion of the vehicle. Third, the used image features are uniformly distributed over the whole image. This benefits twice: In dynamic scenes where most of the detected features lie on independently moving objects, the technique guarantees that not all image features fall on independently moving objects but also on the static background. Second, the bucketing reduces the drift rates of the approach."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "3. We have run SFM on a single camera, which means the scale of translation is unknown. However, as you may have observed, the predicted trajectory is still somehow similar to the ground-truth trajectory. How does $\\mathtt{libviso2}$ handle this ambiguity ($\\mathtt{viso\\_mono.cpp: Line 245}$)?  **(5 points)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "* For SFM on a single camera or monocular vision, $\\mathtt{libviso2}$ uses 8 point algorithm for fundamental matrix computation. For estimating the scale, it assumes that the camera is moving at a known and fixed height over ground. It computes the scaling factor by the ratio of estimated height and actual height (known). $\\mathtt{libviso2}$ scales all the estimations with this computed scaling factor to handle monocular vision scale ambiguity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "4. Briefly explain the RANSAC algorithm used in $\\mathtt{libviso2}$ ($\\mathtt{viso\\_mono.cpp: Line 113 - 129}$).  **(5 points)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "* For \"param.ransac_iters\" times: <br>\n",
    "  – Randomly pick 8 correspondences across two images <br>\n",
    "  – Compute Fundamental Matrix($ F $) using these 8 correspondences <br>\n",
    "  – Among all correspondences, count number of inliers with $x^TFx$ less than 'param.inlier_threshold' <br>\n",
    "\n",
    "  Pick the Fundamental Matrix with the largest number of inliers <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Question 2: Using SIFT [4] for SFM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "In the second task, you are required to use keypoints and feature descriptors from SIFT for SFM. The SIFT implementation can be found in directory $\\mathtt{SIFT}$. \n",
    "\n",
    "(A) Go to $\\mathtt{SIFT}$ directory and run $\\mathtt{runSIFT.py}$ (e.g. `python runSIFT.py --input /datasets/cse152-252-sp20-public/dataset_SfM/sequences/00/image_0/`). You will save the detected keypoints and feature descriptors under the directory $\\mathtt{SIFT}$. For image $\\mathtt{000abc.png}$, the pre-computed features and keypoints should be saved in a $\\mathtt{.npy}$ file named as $\\mathtt{000abc\\_feature.npy}$. The variable should be a $130 \\times N$ matrix with $\\mathtt{single}$ precision, where $N$ is the number of feature points being detected. For each $130$-dimensional feature vector, the first two dimensions are the location of the keypoints (column number first and then row number) on the image plane and the last $128$ dimensions are the feature descriptor. \n",
    "\n",
    "(B) Run the following script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import runFeature\n",
    "dataset_path = '/datasets/cse152-252-sp20-public/dataset_SfM'\n",
    "feature_dir = 'SIFT'\n",
    "errors_df = runFeature.runSFM(dataset_path, feature_dir )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "1. Report the final rotation and translation error. **(2 points)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "* Final Frame Mean Rotation and Mean Translation Error are 0.00243 and 0.55841 respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Not to be reported'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"Not to be reported\"\"\"\n",
    "# print(\"Rotation and Transition Error metric for all frames \\n\")\n",
    "# errors_df.index = np.arange(1, len(errors_df) + 1)\n",
    "# errors_df.index.name = \"Frame Number\"\n",
    "# with pd.option_context('display.max_rows', None, 'display.max_columns', None):\n",
    "#     print(errors_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Next, answer the following questions:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "2. Does SIFT yield higher accuracy than the original $\\mathtt{libviso2}$? Why or why not? If not, can you suggest one possible way to improve? **(5 points)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "* SIFT yields approximately same error/accuracy as $\\mathtt{libviso2}$. $\\mathtt{libviso2}$ uses corners and blob-like keypoints similar to SIFT. In the dataset there is no significant scale and illumination variation thus extracting SIFT keypoints do not give any advantage. Also the motion of car is in a plane thus no advantange of rotation invariance by SIFT keypoints are not advantageous here.\n",
    "\n",
    "* Few Literatures$^1$ have suggested idea to divide the SIFT features extracted from object image into several sub-collections before they are matched. The features are divided into several sub-collections considering the features arising from different octaves, that is from different frequency domains. This has shown significant improvement in accuracy and speed of the feature matching. <br>\n",
    "Some literatures also suggest that Modifiable Area Harmony Dominating Rectification (MHDR) method$^2$ is useful to eliminate mismatched key-point couples automatically and protect matching couples. MHDR method has promising results in improving the accuracy and efficiency of the SIFT image matching. Further, SIFT features matching algorithm can also be improved by changing distance metric from euclidean to other problem specific metrics. \n",
    "\n",
    "  $^1$ = https://dl.acm.org/doi/10.5555/2227536.2227552 <br>\n",
    "  $^2$ = https://link.springer.com/chapter/10.1007/978-3-642-28308-6_31"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "3. Explain how SIFT achieves invariance to \n",
    "       a. illumination\n",
    "       b. rotation\n",
    "       c. scale\n",
    " **(3 points)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "* The SIFT algorithm has following major stages of computation used to generate the set of image features: Scale-space extrema detection, Keypoint localization, Orientation assignment and Keypoint descriptor. <br> \n",
    "  Invariance to each illumination, rotation and scale in relation to these stages are described below:\n",
    "    1. Illumination: At final step of computation, the descriptors obtained are normalized to unit length. This make the descriptors invariant to affine changes in illumination. Robustness to non-linear illumination change is obtained by thresholding the values in the unit feature to each be no larger than 0.2 and then renormalizing to unit.\n",
    "    2. Rotation: At orientation assignment step, each keypoint is assigned one or more orientations based on local image gradient directions. This is the key step in achieving invariance to rotation as the keypoint descriptor can be represented relative to this orientation and therefore achieve invariance to image rotation.\n",
    "    3. Scale: At first step of algorithm, Local extrema detection(keypoints) are searched over all image location in difference of gaussian(DOG) at different scales make the algorithm invariant to scale. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Question 3: Using SuperPoint[1] for SFM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now you are required to use keypoints and feature descriptors from SuperPoint for SFM. The code for the trained model of this method can be found from the $\\mathtt{SuperPoint}$.\n",
    "\n",
    "(A) Go to $\\mathtt{SuperPoint}$ directory and run $\\mathtt{demo\\_superpoint.py}$.`python demo_superpoint.py --input /datasets/cse152-252-sp20-public/dataset_SfM/sequences/00/image_0/ --cuda` The detected keypoints and feature descriptors are under the directory $\\mathtt{SuperPoint}$. The file format is similar to the SIFT case. For image $\\mathtt{000abc.png}$, the pre-computed features and keypoints should be saved in a $\\mathtt{.npy}$ file named as $\\mathtt{000abc\\_feature.npy}$. The variable is a $258\\times N$ matrix with $\\mathtt{single}$ precision, where $N$ is the number of feature points being detected. For each $258$-dimensional feature vector, the first two dimensions are the locations of the keypoint (column number first and then row number) on the image plane and the last $256$ dimensions represent the feature descriptor. \n",
    "\n",
    "(B) Run the following script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import runFeature\n",
    "dataset_path = '/datasets/cse152-252-sp20-public/dataset_SfM'\n",
    "feature_dir = 'SuperPoint'\n",
    "errors_df=runFeature.runSFM(dataset_path, feature_dir )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Report the final rotation and translation error. **(2 points)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Final Frame Mean Rotation and Mean Translation Error are 0.00369 and 0.52405 respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Not to be reported'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"Not to be reported\"\"\"\n",
    "# print(\"Rotation and Transition Error metric for all frames \\n\")\n",
    "# errors_df.index = np.arange(1, len(errors_df) + 1)\n",
    "# errors_df.index.name = \"Frame Number\"\n",
    "# with pd.option_context('display.max_rows', None, 'display.max_columns', None):\n",
    "#     print(errors_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Next, answer the following questions:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "2. Does SuperPoint yield higher accuracy than the original $\\mathtt{libviso2}$? If so, why? If not, what steps can you take to improve? **(5 points)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "* SuperPoint yields approximately same error/accuracy as $\\mathtt{libviso2}$. No improvement is obtained because MagicPoint detected trained on the pseudo-ground truths on synthetic images, do not generalize well to complex images in this dataset. <br>\n",
    "\n",
    "  Improvement: One clear method to improvement is to iterate multiple times to increase impact of self-training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "3. Explain briefly how the following issues are being handled in SuperPoint: **(3 points)**\n",
    "       a. Obtaining ground truth for keypoints.\n",
    "       b. Cheaply obtaining accurate ground truth matches, as compared to LIDAR in UCN or SFM in LIFT.\n",
    "       c. Learning a correlated feature representation for keypoint detection and description? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "1. In SuperPoint, we create a large dataset of pseudo-ground truth interest point locations in real images, supervised by the interest point detector itself, rather than a large-scale human annotation effort.\n",
    "2. MagicPoint detector network is trained on synthetic dataset. we develop a multi-scale, multi-transform technique and Homographic Adaptation to improve its performance on real-world example. These techniques are computationally less expensive and have proven to be effective in improving networks performance. Thus cheaply obtaining accurate ground truths as compared to LIDAR in UCN or SFM in LIFT. <br>\n",
    "3. The network uses the features from same shared encoder for both detection and description."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Question 4: Using SPyNet [5] for SFM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now we will compute camera motion from optical flow computed using SPyNet. We first uniformly sample points in an image, then consider the flow-displaced point in the other image as a match. A modified PyTorch implementation of SPyNet is provided in directory  $\\mathtt{Flow}$.\n",
    "\n",
    "(A) Go to $\\mathtt{Flow}$ and run $\\mathtt{demo\\_spynet.py}$. \n",
    "\n",
    "(B) Run the following script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import runMatch\n",
    "dataset_path = '/datasets/cse152-252-sp20-public/dataset_SfM'\n",
    "feature_dir = 'Flow'\n",
    "errors_df=runMatch.runSFM(dataset_path, feature_dir )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "1. Report the final rotation and translation error. **(2 points)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "* Final Frame Mean Rotation and Mean Translation Error are 0.00372 and 0.79390 respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Not to be reported'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''Not to be reported'''\n",
    "# print(\"Rotation and Transition Error metric for all frames \\n\")\n",
    "# errors_df.index = np.arange(1, len(errors_df) + 1)\n",
    "# errors_df.index.name = \"Frame Number\"\n",
    "# with pd.option_context('display.max_rows', None, 'display.max_columns', None):\n",
    "#     print(errors_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Next, answer the following questions:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "2. Does SPyNet yield higher accuracy than the original $\\mathtt{libviso2}$? Why or why not? If not, what steps can you take to improve? **(5 points)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "* No. SPyNet yields lower accuracy than libviso2. SPyNet uniformly samples points in the image, then consider the flow-displaced point in the other image as a match. By Uniform sampling it tries to match features which are not interest points. This leds to mismatching and thus reduces the accuracy. <br>\n",
    "  One of the ways to improve the algorithm is to sample interest points rather than uniform sampling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "3. Explain how SPyNet achieves accurate flow with significantly lower computational cost. **(3 points)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "* SPyNet scales the image down and creates image pyramid of different resolution level. The Network computes difference between true and upsampled flows at each level called residual field. The residual fields across the image at a certain level are usually a small displacements and their computation is relatively simple. Thus, each level solves a simple problem of finding these residual fields, requiring a small CNN network with far less trainable parameters than networks like FlowNet. Further rather than having a network for warping, SPyNet bilinearly interpolates image flow as consecutive levels."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Question 5: Using Sfm-learner for SFM\n",
    "Now we will use deep neural networks for SFM. Please follow the open-source Sfmlearner repository (https://github.com/ClementPinard/SfmLearner-Pytorch) and the paper (https://people.eecs.berkeley.edu/~tinghuiz/projects/SfMLearner/cvpr17_sfm_final.pdf) to answer the following questions.\n",
    "\n",
    "1. Follow the steps to add sfm-learner as a submodule:\n",
    "`git submodule add https://github.com/ClementPinard/SfmLearner-Pytorch.git`.\n",
    "1. you might have to do `git submodule update --init --recursive` for older git version or on other systems which already have previous commits locally.\n",
    "1. For cloning the repository use `git clone --recursive <project url>` to download submodule also"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "1. Describe how photometric loss is implemented in Sfmlearner? Please refer to the paper and the code. **(5 points)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "* The Photometric loss is based on warping nearby views to the target using the computed depth and pose.<br>\n",
    "  The depth and explainability mask predictions from single-view depth network and pose/explainability network are used to compute the photometric loss. The source image is warped and L1 loss is computed with respected to target image. The loss is computed at different scale(factor of 2) of image and is accumulated over each scales. <br>\n",
    "  Mathematically loss can be written as: <br>\n",
    "  Photometric Loss $(L_{1}) = \\displaystyle\\sum_{<I_1, ... ,I_N> \\in S} \\sum_p \\hat{E}_s(p) \\mid I_t(p) - \\hat{I_s}(p)  \\mid$ <br>\n",
    "  where, $\\hat{E}_s(p)$ is output(prediction) of explainability prediction network and is per-pixel($p$) soft mask for each target-source pair.\n",
    "\n",
    "  Further, to encourage non-trivial(non-zero) prediction of $\\hat{E}_s$, the author introduce a regularization loss called explainability loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "2. Train the model and show the training curve, validation curve, and visualization of predicted depth and warped images? Which loss is decreasing and which is increasing? Why? **(10 points)**\n",
    "\n",
    " Visualize the specific images: `val_Depth_Output_Normalized/0`, `val_Warped_Outputs/0` from tensorboard.\n",
    " \n",
    " Training data: \n",
    "  `/datasets/cse152-252-sp20-public/sfmlearner_h128w416`.\n",
    "\n",
    "  #### Install the environment\n",
    "  `pip install -r requirements.txt`\n",
    "  - fix scipy version problem: \n",
    "\n",
    "  `pip install scipy==1.1.0 --user`\n",
    "\n",
    "  #### Training script\n",
    "  `cd SfmLearner-Pytorch`\n",
    "\n",
    "  `python3 train.py /datasets/cse152-252-sp20-public/sfmlearner_h128w416 -b4 -m0.2 -s0.1 --epoch-size 3000 --sequence-length 3 --log-output`\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "All the plots are with smoothness factor of 0.4\n",
    "\n",
    "**Loss vs epochs Plots for Training Dataset:**\n",
    "\n",
    "<figure>\n",
    "<figcaption>Total Loss</figcaption>\n",
    "<img src=\"pyviso/SfmLearner-Pytorch/results_sfmlearner/3losses/total_loss.svg\" style=\"width:300px;\"/>\n",
    "<figure/>\n",
    "\n",
    "<figure>\n",
    "<figcaption>Disparity Smoothness Loss</figcaption>\n",
    "<img src=\"pyviso/SfmLearner-Pytorch/results_sfmlearner/3losses/disparity_smoothness_loss.svg\" style=\"width:300px;\"/>\n",
    "<figure/>\n",
    "    \n",
    "<figure>\n",
    "<figcaption>Explanability Loss</figcaption>\n",
    "<img src=\"pyviso/SfmLearner-Pytorch/results_sfmlearner/3losses/explanability_loss.svg\" style=\"width:300px;\"/>\n",
    "<figure/>\n",
    "\n",
    "<figure>\n",
    "<figcaption>Photometric Error</figcaption>\n",
    "<img src=\"pyviso/SfmLearner-Pytorch/results_sfmlearner/3losses/photometric_error.svg\" style=\"width:300px;\"/>\n",
    "<figure/>\n",
    "\n",
    "\n",
    "**Loss vs epochs Plots for Validation Dataset:**\n",
    "\n",
    "<figure>\n",
    "<figcaption>Total Validation Loss</figcaption>\n",
    "<img src=\"pyviso/SfmLearner-Pytorch/results_sfmlearner/3losses/Validation_Total_loss.svg\" style=\"width:300px;\"/>\n",
    "<figure/>\n",
    "\n",
    "    \n",
    "<figure>\n",
    "<figcaption>Explanability Loss</figcaption>\n",
    "<img src=\"pyviso/SfmLearner-Pytorch/results_sfmlearner/3losses/Validation_Exp_loss.svg\" style=\"width:300px;\"/>\n",
    "<figure/>\n",
    "\n",
    "<figure>\n",
    "<figcaption>Photometric Error</figcaption>\n",
    "<img src=\"pyviso/SfmLearner-Pytorch/results_sfmlearner/3losses/Validation_Photo_loss.svg\" style=\"width:300px;\"/>\n",
    "<figure/>\n",
    "\n",
    "**Visualization of Results**\n",
    "<figure>\n",
    "<figcaption>val_Depth_Output_Normalized/0</figcaption>\n",
    "<img src=\"pyviso/SfmLearner-Pytorch/results_sfmlearner/3losses/val_Depth_Output_Normalized_0.png\" style=\"width:500px;\"/>\n",
    "<figure/>\n",
    "\n",
    "<figure>\n",
    "<figcaption>val_Warped_Outputs/0</figcaption>\n",
    "<img src=\"pyviso/SfmLearner-Pytorch/results_sfmlearner/3losses/val_Warped_Outputs__0.png\" style=\"width:500px;\"/>\n",
    "<figure/>\n",
    "    \n",
    "    \n",
    "    \n",
    "**Discussion on Losses**  \n",
    "Total Loss, Photometric Loss and Explanability Loss are decreasing whereas disaprity smoothness loss is increasing. This is because optimal solutions of the total loss is opposite direction to the smoothness loss term. Note that these curves will be affected by the weights of each term in total loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "3. When training the model, we use 3 consecutive frames. Now, you will use the photometric consistency between the 1st and the 3rd frame. To be more specific, you can get the pose $T_{1,3} = T_{2,3} @ T_{1,2}$, where $T_{1,2}, T_{2,3}$ have already been computed in the original code. Add the constraint to the total loss and report the results in the same manner as in part 2 above. **(10 points)**\n",
    "\n",
    "  #### Training script\n",
    "  `cd SfmLearner-Pytorch`\n",
    "\n",
    "  `python3 train.py /datasets/cse152-252-sp20-public/sfmlearner_h128w416 -b4 -m0.2 -s0.1 --epoch-size 3000 --sequence-length 3 --log-output --with-photocon-loss`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "``answer here``\n",
    "\n",
    "All the plots are with smoothness factor of 0.4\n",
    "\n",
    "**Loss vs epochs Plots for Training Dataset:**\n",
    "\n",
    "<figure>\n",
    "<figcaption>Total Loss</figcaption>\n",
    "<img src=\"pyviso/SfmLearner-Pytorch/results_sfmlearner/4losses1/total_loss.svg\" style=\"width:300px;\"/>\n",
    "<figure/>\n",
    "\n",
    "<figure>\n",
    "<figcaption>Disparity Smoothness Loss</figcaption>\n",
    "<img src=\"pyviso/SfmLearner-Pytorch/results_sfmlearner/4losses1/disparity_smoothness_loss.svg\" style=\"width:300px;\"/>\n",
    "<figure/>\n",
    "    \n",
    "<figure>\n",
    "<figcaption>Explanability Loss</figcaption>\n",
    "<img src=\"pyviso/SfmLearner-Pytorch/results_sfmlearner/4losses1/explanability_loss.svg\" style=\"width:300px;\"/>\n",
    "<figure/>\n",
    "\n",
    "<figure>\n",
    "<figcaption>Photometric Error</figcaption>\n",
    "<img src=\"pyviso/SfmLearner-Pytorch/results_sfmlearner/4losses1/photometric_error.svg\" style=\"width:300px;\"/>\n",
    "<figure/>\n",
    "\n",
    "\n",
    "**Loss vs epochs Plots for Validation Dataset:**\n",
    "\n",
    "<figure>\n",
    "<figcaption>Total Validation Loss</figcaption>\n",
    "<img src=\"pyviso/SfmLearner-Pytorch/results_sfmlearner/4losses1/Validation_Total_loss.svg\" style=\"width:300px;\"/>\n",
    "<figure/>\n",
    "\n",
    "    \n",
    "<figure>\n",
    "<figcaption>Explanability Loss</figcaption>\n",
    "<img src=\"pyviso/SfmLearner-Pytorch/results_sfmlearner/4losses1/Validation_Exp_loss.svg\" style=\"width:300px;\"/>\n",
    "<figure/>\n",
    "\n",
    "<figure>\n",
    "<figcaption>Photometric Error</figcaption>\n",
    "<img src=\"pyviso/SfmLearner-Pytorch/results_sfmlearner/4losses1/Validation_Photo_loss.svg\" style=\"width:300px;\"/>\n",
    "<figure/>\n",
    "\n",
    "**Visualization of Results**\n",
    "<figure>\n",
    "<figcaption>val_Depth_Output_Normalized/0</figcaption>\n",
    "<img src=\"pyviso/SfmLearner-Pytorch/results_sfmlearner/4losses1/val_Depth_Output_Normalized_0.png\" style=\"width:500px;\"/>\n",
    "<figure/>\n",
    "\n",
    "<figure>\n",
    "<figcaption>val_Warped_Outputs/0</figcaption>\n",
    "<img src=\"pyviso/SfmLearner-Pytorch/results_sfmlearner/4losses1/val_Warped_Outputs__0.png\" style=\"width:500px;\"/>\n",
    "<figure/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "4. Now, you will evaluate your models from parts (2), (3) on the KITTI odometry dataset. What are the error metrics used in Sfm-learner? Please report the `ATE` and `RE` for sequence `09` and `10`. **(5 points)**\n",
    "\n",
    "  Data: `/datasets/cse152-252-sp20-public/kitti`.\n",
    "\n",
    "  #### Evaluation script\n",
    "  `python3 test_pose.py /path/to/posenet --dataset-dir /datasets/cse152-252-sp20-public/kitti --sequences 09`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "**Error metrics used:** Absolute Translation Error (ATE) and Rotational Error (RE)\n",
    "\n",
    "**Simple SfmLearner Model(3 losses) after training for 200 epochs**\n",
    "* Performance on sequence 09:\n",
    "\n",
    "|       | ATE  |  RE  |\n",
    "| ---   | ---  | ---  |\n",
    "| mean  |0.0119|0.0023|\n",
    "| std   |0.0065|0.0012|\n",
    "\n",
    "* Performance on sequence 10:\n",
    "\n",
    "|          |     ATE  |     RE    |\n",
    "|   ---    |    --    |     --    |\n",
    "|mean      |   0.0108 |    0.0024 |\n",
    "|std       |  0.0084  |   0.0016  |\n",
    "\n",
    "**SfmLearner Model with Photoconsistency Loss(4 losses) after training for 200 epochs**\n",
    "* Performance on sequence 09: \n",
    "\n",
    "|          |  ATE   |   RE    |\n",
    "|  ---     |  --    |   --    |\n",
    "|   mean   | 0.0128 |   0.0024|\n",
    "|   std    | 0.0074 |   0.0013|\n",
    "\n",
    "* Performance on sequence 10:\n",
    "\n",
    "|          |   ATE    |    RE   |\n",
    "|   --     |    --    |   --    |\n",
    "|mean      |   0.0098 |  0.0025 |\n",
    "|std       |   0.0071 |  0.0017 |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Notes:\n",
    "- scp data to local machines:\n",
    "\n",
    "`scp -r <USERNAME>@dsmlp-login.ucsd.edu:/datasets/cse152-252-sp20-public/sfmlearner_h128w416.zip`\n",
    "\n",
    "`scp -r <USERNAME>@dsmlp-login.ucsd.edu:/datasets/cse152-252-sp20-public/kitti.zip`\n",
    "...\n",
    "- tensorboard: open jupyter notebook from the link after `launch-scipy-ml-gpu.sh`. Click new `Tensorboard ..`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# References\n",
    "1. Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 224–236, 2018.\n",
    "2. Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.\n",
    "3. Andreas Geiger, Julius Ziegler, and Christoph Stiller. Stereoscan: Dense 3d reconstruction in real-time. In Intelligent Vehicles Symposium (IV), 2011.\n",
    "4. David G Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 60(2):91–110, 2004.\n",
    "5. Anurag Ranjan and Michael J Black. Optical flow estimation using a spatial pyramid network. In\n",
    "Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4161–4170, 2017.\n",
    "6. A. Vedaldi and B. Fulkerson. VLFeat: An open and portable library of computer vision algorithms. http://www.vlfeat.org/, 2008.\n",
    "7. Lucas, Bruce D., and Takeo Kanade. \"An iterative image registration technique with an application to stereo vision.\" (1981): 674.\n",
    "8. B. Kitt, A. Geiger, and H. Lategahn, “Visual odometry based on stereo image sequences with RANSAC-based outlier rejection scheme,” in 2010 IEEE Intelligent Vehicles Symposium, La Jolla, CA, USA, Jun. 2010, pp. 486–492, doi: 10.1109/IVS.2010.5548123.\n",
    "9. A. Geiger, J. Ziegler, and C. Stiller, “StereoScan: Dense 3d reconstruction in real-time,” in 2011 IEEE Intelligent Vehicles Symposium (IV), Baden-Baden, Germany, Jun. 2011, pp. 963–968, doi: 10.1109/IVS.2011.5940405."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4rc1"
  },
  "nbdime-conflicts": {
   "local_diff": [
    {
     "key": "kernelspec",
     "op": "add",
     "value": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
     }
    },
    {
     "key": "language_info",
     "op": "add",
     "value": {
      "codemirror_mode": {
       "name": "ipython",
       "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.10"
     }
    }
   ],
   "remote_diff": [
    {
     "key": "kernelspec",
     "op": "add",
     "value": {
      "display_name": "py36-torch",
      "language": "python",
      "name": "myenv"
     }
    },
    {
     "key": "language_info",
     "op": "add",
     "value": {
      "codemirror_mode": {
       "name": "ipython",
       "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.8"
     }
    }
   ]
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}