c03-data-science-methodology/res/DS0103EN-Review-From-Requirements-to-Collection.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "<img src = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DS0103EN-SkillsNetwork/labs/Module%202/images/IDSNlogo.png\" width = 400>\n",
        "\n",
        "# From Requirements to Collection\n",
        "\n",
        "Estimated time needed: **15** minutes\n",
        "\n",
        "## Objectives\n",
        "\n",
        "After completing this lab you will be able to:\n",
        "\n",
        "*   Understand Data Requirements\n",
        "*   Explore the stages in Data Collection\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "## Table of Contents\n",
        "\n",
        "<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n",
        "\n",
        "1.  [Data Requirements](#0)<br>\n",
        "2.  [Data Collection](#2)<br>\n",
        "\n",
        "</div>\n",
        "<hr>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "# Data Requirements <a id=\"0\"></a>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "<img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DS0103EN-SkillsNetwork/labs/Module%202/images/lab2_fig1_flowchart_data_requirements.png\" width=500>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "In the videos, we learned that the chosen analytic approach determines the data requirements. Specifically, the analytic methods to be used require certain data content, formats and representations, guided by domain knowledge.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "In the **From Problem to Approach Lab**, we determined that automating the process of determining the cuisine of a given recipe or dish is potentially possible using the ingredients of the recipe or the dish. In order to build a model, we need extensive data of different cuisines and recipes.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "Identifying the required data fulfills the data requirements stage of the data science methodology.\n",
        "\n",
        "***\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "# Data Collection <a id=\"2\"></a>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "<img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DS0103EN-SkillsNetwork/labs/Module%202/images/lab2_fig2_flowchart_data_collection.png\" width=500>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "In the initial data collection stage, data scientists identify and gather the available data resources. These can be in the form of structured, unstructured, and even semi-structured data relevant to the problem domain.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "#### Web Scraping of Online Food Recipes\n",
        "\n",
        "A researcher named Yong-Yeol Ahn scraped tens of thousands of food recipes (cuisines and ingredients) from three different websites, namely:\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "<img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DS0103EN-SkillsNetwork/labs/Module%202/images/lab2_fig3_allrecipes.png\" width=500>\n",
        "<div align=\"center\">\n",
        "www.allrecipes.com\n",
        "</div>\n",
        "<br/><br/>\n",
        "<img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DS0103EN-SkillsNetwork/labs/Module%202/images/lab2_fig4_epicurious.png\" width=500>\n",
        "<div align=\"center\">\n",
        "www.epicurious.com\n",
        "</div>\n",
        "<br/><br/>\n",
        "<img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DS0103EN-SkillsNetwork/labs/Module%202/images/lab2_fig5_menupan.png\" width=500>\n",
        "<div align=\"center\">\n",
        "www.menupan.com\n",
        "</div>\n",
        "<br/><br/>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "For more information on Yong-Yeol Ahn and his research, you can read his paper on [Flavor Network and the Principles of Food Pairing](http://yongyeol.com/papers/ahn-flavornet-2011.pdf?utm_medium=Exinfluencer&utm_source=Nurture&utm_content=000026UJ&utm_term=10006555&utm_id=SkillsNetwork-Courses-IBMDeveloperSkillsNetwork-DS0103EN-SkillsNetwork-20083987&utm_email=Email&utm_campaign=PLACEHOLDER).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "Luckily, we will not need to carry out any data collection as the data that we need to meet the goal defined in the business understanding stage is readily available.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "#### We have already acquired the data and placed it on an IBM server. Let's download the data and take a look at it.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "<strong>Important note:</strong> Please note that you are not expected to know how to program in python. The following code is meant to illustrate the stage of data collection, so it is totally fine if you do not understand the individual lines of code. There will be a full course in this certificate on programming in python, <a href=\"http://cocl.us/PY0101EN_DS0103EN_LAB2_PYTHON_edX?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDS0103ENSkillsNetwork20083987-2022-01-01\">Python for Data Science</a>, which will teach you how to program in Python if you decide to complete this certificate.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "### Using this notebook:\n",
        "\n",
        "To run any of the following cells of code, you can type **Shift + Enter** to excute the code in a cell.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "Get the version of Python installed.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "outputs": [],
      "source": [
        "# check Python version\n",
        "# !python -V"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "Read the data from the IBM server into a *pandas* dataframe.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "tags": []
      },
      "outputs": [],
      "source": [
        "import pandas as pd # download library to read data into dataframe\n",
        "pd.set_option('display.max_columns', None)\n",
        "\n",
        "# Uncomment if running locally, else download data using the following code cell\n",
        "# recipes = pd.read_csv(\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DS0103EN-SkillsNetwork/labs/Module%202/recipes.csv\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import piplite\n",
        "await piplite.install(['skillsnetwork'])\n",
        "import skillsnetwork\n",
        "\n",
        "# Download to current directory with same filename\n",
        "await skillsnetwork.download_dataset(\n",
        "    \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DS0103EN-SkillsNetwork/labs/Module%202/recipes.csv\")\n",
        "recipes = pd.read_csv(\"recipes.csv\")\n",
        "print(\"Data read into dataframe!\") # takes about 30 seconds"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "Show the first few rows.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "tags": []
      },
      "outputs": [],
      "source": [
        "recipes.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "Get the dimensions of the dataframe.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "tags": []
      },
      "outputs": [],
      "source": [
        "recipes.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "So our dataset consists of 57,691 recipes. Each row represents a recipe, and for each recipe, the corresponding cuisine is documented as well as whether 384 ingredients exist in the recipe or not beginning with almond and ending with zucchini.\n",
        "\n",
        "***\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "Now that the data collection stage is complete, data scientists typically use descriptive statistics and visualization techniques to better understand the data and get acquainted with it. Data scientists, essentially, explore the data to:\n",
        "\n",
        "*   understand its content,\n",
        "*   assess its quality,\n",
        "*   discover any interesting preliminary insights, and,\n",
        "*   determine whether additional data is necessary to fill any gaps in the data.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        }
      },
      "source": [
        "### Thank you for completing this lab!\n",
        "\n",
        "This notebook is part of a course called *The Data Science Method*. If you accessed this notebook outside the course, you can take this course, online by clicking [here](https://cocl.us/DS0103EN-Review-From-Requirements-to-Collection).\n",
        "\n",
        "## Author\n",
        "\n",
        "<a href=\"https://www.linkedin.com/in/aklson/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDS0103ENSkillsNetwork20083987-2022-01-01\" target=\"_blank\">Alex Aklson</a>\n",
        "\n",
        "## Change Log\n",
        "\n",
        "| Date (YYYY-MM-DD) | Version | Changed By | Change Description                 |\n",
        "| ----------------- | ------- | ---------- | ---------------------------------- |\n",
        "| 2021-04-06        | 2.1     | Malika     | Updated lab link                   |\n",
        "| 2020-09-25        | 2.0     | Lakshmi    | Fixed Typo errors                  |\n",
        "| 2020-08-27        | 2.0     | Lavanya    | Moved lab to course repo in GitLab |\n",
        "\n",
        "<hr>\n",
        "\n",
        "## <h3 align=\"center\"> © IBM Corporation 2020. All rights reserved. <h3/>\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.9"
    },
    "widgets": {
      "state": {},
      "version": "1.1.2"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}