9.4 Removing Duplicate Entries.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](img/cover4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><em>Copyright! This material is protected, please do not copy or distribute.    by:Taher Assaf</em></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "<h1 align=\"center\">Udemy course : Python Bootcamp for Data Science 2021 Numpy Pandas & Seaborn</h1> \n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9.4 Removing Duplicate Entries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we import numpy and pandas libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we read a dataframe with **duplicate rows** using the function **pd.read_csv()**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "hide_input": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>AtBat</th>\n",
       "      <th>Hits</th>\n",
       "      <th>HmRun</th>\n",
       "      <th>Runs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Andy Allanson</td>\n",
       "      <td>293</td>\n",
       "      <td>66</td>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alan Ashby</td>\n",
       "      <td>315</td>\n",
       "      <td>81</td>\n",
       "      <td>7</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Alvin Davis</td>\n",
       "      <td>479</td>\n",
       "      <td>130</td>\n",
       "      <td>18</td>\n",
       "      <td>66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Andy Allanson</td>\n",
       "      <td>293</td>\n",
       "      <td>66</td>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Andre Dawson</td>\n",
       "      <td>496</td>\n",
       "      <td>141</td>\n",
       "      <td>20</td>\n",
       "      <td>65</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Name  AtBat  Hits  HmRun  Runs\n",
       "0  Andy Allanson    293    66      1    30\n",
       "1     Alan Ashby    315    81      7    24\n",
       "2    Alvin Davis    479   130     18    66\n",
       "3  Andy Allanson    293    66      1    30\n",
       "4   Andre Dawson    496   141     20    65"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt1 = pd.read_csv('data/ex1.csv')\n",
    "dt1.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To check for duplicate entries we use the function **duplicated()**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "hide_input": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     False\n",
       "1     False\n",
       "2     False\n",
       "3      True\n",
       "4     False\n",
       "5     False\n",
       "6     False\n",
       "7      True\n",
       "8     False\n",
       "9      True\n",
       "10    False\n",
       "11    False\n",
       "12    False\n",
       "13    False\n",
       "14    False\n",
       "15    False\n",
       "16    False\n",
       "17    False\n",
       "18    False\n",
       "19    False\n",
       "20    False\n",
       "21    False\n",
       "22    False\n",
       "23    False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt1.duplicated()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can add the function **any()** to the prevous code to return a single boolean value to check for duplicates in the entire dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "hide_input": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt1.duplicated().any()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "WE can delete the duplicate entries using the function **drop_duplicates()**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "hide_input": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>AtBat</th>\n",
       "      <th>Hits</th>\n",
       "      <th>HmRun</th>\n",
       "      <th>Runs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Andy Allanson</td>\n",
       "      <td>293</td>\n",
       "      <td>66</td>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alan Ashby</td>\n",
       "      <td>315</td>\n",
       "      <td>81</td>\n",
       "      <td>7</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Alvin Davis</td>\n",
       "      <td>479</td>\n",
       "      <td>130</td>\n",
       "      <td>18</td>\n",
       "      <td>66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Andre Dawson</td>\n",
       "      <td>496</td>\n",
       "      <td>141</td>\n",
       "      <td>20</td>\n",
       "      <td>65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Andres Galarraga</td>\n",
       "      <td>321</td>\n",
       "      <td>87</td>\n",
       "      <td>10</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Alfredo Griffin</td>\n",
       "      <td>594</td>\n",
       "      <td>169</td>\n",
       "      <td>4</td>\n",
       "      <td>74</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Al Newman</td>\n",
       "      <td>185</td>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Argenis Salazar</td>\n",
       "      <td>298</td>\n",
       "      <td>73</td>\n",
       "      <td>0</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Andres Thomas</td>\n",
       "      <td>323</td>\n",
       "      <td>81</td>\n",
       "      <td>6</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Andre Thornton</td>\n",
       "      <td>401</td>\n",
       "      <td>92</td>\n",
       "      <td>17</td>\n",
       "      <td>49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Alan Trammell</td>\n",
       "      <td>574</td>\n",
       "      <td>159</td>\n",
       "      <td>21</td>\n",
       "      <td>107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Alex Trevino</td>\n",
       "      <td>202</td>\n",
       "      <td>53</td>\n",
       "      <td>4</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Andy VanSlyke</td>\n",
       "      <td>418</td>\n",
       "      <td>113</td>\n",
       "      <td>13</td>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Alan Wiggins</td>\n",
       "      <td>239</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Bill Almon</td>\n",
       "      <td>196</td>\n",
       "      <td>43</td>\n",
       "      <td>7</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Billy Beane</td>\n",
       "      <td>183</td>\n",
       "      <td>39</td>\n",
       "      <td>3</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Buddy Bell</td>\n",
       "      <td>568</td>\n",
       "      <td>158</td>\n",
       "      <td>20</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Buddy Biancalana</td>\n",
       "      <td>190</td>\n",
       "      <td>46</td>\n",
       "      <td>2</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Bruce Bochte</td>\n",
       "      <td>407</td>\n",
       "      <td>104</td>\n",
       "      <td>6</td>\n",
       "      <td>57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Bruce Bochy</td>\n",
       "      <td>127</td>\n",
       "      <td>32</td>\n",
       "      <td>8</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Barry Bonds</td>\n",
       "      <td>413</td>\n",
       "      <td>92</td>\n",
       "      <td>16</td>\n",
       "      <td>72</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Name  AtBat  Hits  HmRun  Runs\n",
       "0      Andy Allanson    293    66      1    30\n",
       "1         Alan Ashby    315    81      7    24\n",
       "2        Alvin Davis    479   130     18    66\n",
       "4       Andre Dawson    496   141     20    65\n",
       "5   Andres Galarraga    321    87     10    39\n",
       "6    Alfredo Griffin    594   169      4    74\n",
       "8          Al Newman    185    37      1    23\n",
       "10   Argenis Salazar    298    73      0    24\n",
       "11     Andres Thomas    323    81      6    26\n",
       "12    Andre Thornton    401    92     17    49\n",
       "13     Alan Trammell    574   159     21   107\n",
       "14      Alex Trevino    202    53      4    31\n",
       "15     Andy VanSlyke    418   113     13    48\n",
       "16      Alan Wiggins    239    60      0    30\n",
       "17        Bill Almon    196    43      7    29\n",
       "18       Billy Beane    183    39      3    20\n",
       "19        Buddy Bell    568   158     20    89\n",
       "20  Buddy Biancalana    190    46      2    24\n",
       "21      Bruce Bochte    407   104      6    57\n",
       "22       Bruce Bochy    127    32      8    16\n",
       "23       Barry Bonds    413    92     16    72"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt1.drop_duplicates()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make the changes reflected in the original dataframe, we use the argument **inplace = True**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "hide_input": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>AtBat</th>\n",
       "      <th>Hits</th>\n",
       "      <th>HmRun</th>\n",
       "      <th>Runs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Andy Allanson</td>\n",
       "      <td>293</td>\n",
       "      <td>66</td>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alan Ashby</td>\n",
       "      <td>315</td>\n",
       "      <td>81</td>\n",
       "      <td>7</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Alvin Davis</td>\n",
       "      <td>479</td>\n",
       "      <td>130</td>\n",
       "      <td>18</td>\n",
       "      <td>66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Andre Dawson</td>\n",
       "      <td>496</td>\n",
       "      <td>141</td>\n",
       "      <td>20</td>\n",
       "      <td>65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Andres Galarraga</td>\n",
       "      <td>321</td>\n",
       "      <td>87</td>\n",
       "      <td>10</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Alfredo Griffin</td>\n",
       "      <td>594</td>\n",
       "      <td>169</td>\n",
       "      <td>4</td>\n",
       "      <td>74</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Al Newman</td>\n",
       "      <td>185</td>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Argenis Salazar</td>\n",
       "      <td>298</td>\n",
       "      <td>73</td>\n",
       "      <td>0</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Andres Thomas</td>\n",
       "      <td>323</td>\n",
       "      <td>81</td>\n",
       "      <td>6</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Andre Thornton</td>\n",
       "      <td>401</td>\n",
       "      <td>92</td>\n",
       "      <td>17</td>\n",
       "      <td>49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Alan Trammell</td>\n",
       "      <td>574</td>\n",
       "      <td>159</td>\n",
       "      <td>21</td>\n",
       "      <td>107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Alex Trevino</td>\n",
       "      <td>202</td>\n",
       "      <td>53</td>\n",
       "      <td>4</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Andy VanSlyke</td>\n",
       "      <td>418</td>\n",
       "      <td>113</td>\n",
       "      <td>13</td>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Alan Wiggins</td>\n",
       "      <td>239</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Bill Almon</td>\n",
       "      <td>196</td>\n",
       "      <td>43</td>\n",
       "      <td>7</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Billy Beane</td>\n",
       "      <td>183</td>\n",
       "      <td>39</td>\n",
       "      <td>3</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Buddy Bell</td>\n",
       "      <td>568</td>\n",
       "      <td>158</td>\n",
       "      <td>20</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Buddy Biancalana</td>\n",
       "      <td>190</td>\n",
       "      <td>46</td>\n",
       "      <td>2</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Bruce Bochte</td>\n",
       "      <td>407</td>\n",
       "      <td>104</td>\n",
       "      <td>6</td>\n",
       "      <td>57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Bruce Bochy</td>\n",
       "      <td>127</td>\n",
       "      <td>32</td>\n",
       "      <td>8</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Barry Bonds</td>\n",
       "      <td>413</td>\n",
       "      <td>92</td>\n",
       "      <td>16</td>\n",
       "      <td>72</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Name  AtBat  Hits  HmRun  Runs\n",
       "0      Andy Allanson    293    66      1    30\n",
       "1         Alan Ashby    315    81      7    24\n",
       "2        Alvin Davis    479   130     18    66\n",
       "4       Andre Dawson    496   141     20    65\n",
       "5   Andres Galarraga    321    87     10    39\n",
       "6    Alfredo Griffin    594   169      4    74\n",
       "8          Al Newman    185    37      1    23\n",
       "10   Argenis Salazar    298    73      0    24\n",
       "11     Andres Thomas    323    81      6    26\n",
       "12    Andre Thornton    401    92     17    49\n",
       "13     Alan Trammell    574   159     21   107\n",
       "14      Alex Trevino    202    53      4    31\n",
       "15     Andy VanSlyke    418   113     13    48\n",
       "16      Alan Wiggins    239    60      0    30\n",
       "17        Bill Almon    196    43      7    29\n",
       "18       Billy Beane    183    39      3    20\n",
       "19        Buddy Bell    568   158     20    89\n",
       "20  Buddy Biancalana    190    46      2    24\n",
       "21      Bruce Bochte    407   104      6    57\n",
       "22       Bruce Bochy    127    32      8    16\n",
       "23       Barry Bonds    413    92     16    72"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt1.drop_duplicates(inplace =True)\n",
    "dt1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "<h1 align=\"center\">Thank You</h1> \n",
    "\n",
    "***"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}