electricity_datasheet_lesson1.a.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hello! My name is Taylor Ennen, and I'm a Data Engineer.\n",
    "### This is for you. The Reader. Strong start?\n",
    "### Here's my [twitter](https://www.twitter.com/taylorennen), [github](https://www.github.com/taylor-ennen).\n",
    "### \n",
    "\n",
    "\n",
    "### It's great to have you along as we explore, the world that is.\n",
    "\n",
    "#### This series will be a working demonstration of the skills and tools taught to me through my internship, juniorhood, and resource pooling. If you feel I'm missing something or have ideas on how this guide can be improved, please reach out with those recommendations.\n",
    "\n",
    "#### I mention my current Experience as this may not be applicable to you, and well, here it is. *Alternatively*, this is an opportunity to *NOT*—I repeat, **_NOT_**—let yourself be trapped in comparison to your neighbor, k?, or anyone else. My goal is that each reader will leave with gained knowledge, guided by their present experience.\n",
    "\n",
    "##### Enough of the introduction, or whatever that was.\n",
    "\n",
    "##### Welcome to [INSERT long ambiguous name for what _this_ IS]!!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### The first thing I'm doing to help growth in this series is a recommendation, start by: Reading or Listening, to [Atomic Habits](https://www.amazon.com/Atomic-Habits-Proven-Build-Break/dp/0735211299?&_encoding=UTF8&tag=skylinedevstu-20&linkCode=ur2&linkId=9191ec3810b8ae6321da7ebb29d32d1f&camp=1789&creative=9325\"). \n",
    "###### Disclaimer: I'm using an affiliate link 👍 \n",
    "##### That's it. Lesson over, see you on the next one. Realistically, do it. I chose the audiobook and folded laundry for a few hours and then cleaned up after that. Change. If you have read it, then you know. Great.\n",
    "\n",
    "##### Moving on to item number two; choose a csv with meaningful data.\n",
    "##### #todo: Add CSV stats, rows, columns, #rows #columns, #style of data? -- I would like to present the shape of the CSV perhaps in a miniseries of stylized views advancing in complexity shown from square to knowledge graph by sectors.\n",
    "##### The CSV that I chose to start with is simply named electricity.csv. Contained inside the *\"electricity\"* CSV is a Dataset listing Utility Companies that provide Electricity to Residential, Commercial, and Industrial retail customers, alongside metrics varying from source, use case, revenue, and retail details.\n",
    "\n",
    "##### You can download that dataset yourself by following this link:\n",
    "##### >>>>>>>>[Download the electricity.csv dataset here](https://corgis-edu.github.io/corgis/csv/electricity/)\n",
    "\n",
    "##### Throughout the code you will extensive commenting, It's designed with deeper experience in mind. If they feel like too much, read through twice; without the comments, then with the comments.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: requests in c:\\users\\taylor\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python39\\site-packages (2.31.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\taylor\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python39\\site-packages (from requests) (2023.5.7)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\taylor\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python39\\site-packages (from requests) (3.1.0)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\taylor\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python39\\site-packages (from requests) (2.0.3)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\taylor\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python39\\site-packages (from requests) (3.4)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: You are using pip version 22.0.4; however, version 23.2.1 is available.\n",
      "You should consider upgrading via the 'C:\\Users\\Taylor\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\python.exe -m pip install --upgrade pip' command.\n"
     ]
    }
   ],
   "source": [
    "!pip install requests\n",
    " # This will install the requests library to the Jupyter Notebook environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will import the `requests` library to the *current environment* we are *operating* within."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://corgis-edu.github.io/corgis/datasets/csv/electricity/electricity.csv\n",
      "<class 'str'>\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "# let's set the variable \"url\" to the url string of the data we want to access\n",
    "url=\"https://corgis-edu.github.io/corgis/datasets/csv/electricity/electricity.csv\"\n",
    "print(url)\n",
    "print(type(url))\n",
    "\n",
    "#Awesome! We have the url of the data we want to access. Now we need to access it.\n",
    "#and yes I know we didn't use the `json` library. We'll get to that later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can utilize the `requests` library to access the data from the `url` VARIABLE with a `requests.get` FUNCTION, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(url) \n",
    "assert response \n",
    "\n",
    "# LEARN MORE\n",
    "# this assert keyword says \"validate that what comes after me is true\"\n",
    "# so what we're saying is \"validate that the request we made returned as a successful response\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200\n"
     ]
    }
   ],
   "source": [
    "#let's check if the request was successful\n",
    "\n",
    "print(response.status_code)\n",
    "\n",
    "# LEARN MORE\n",
    "# if the status code is 200, then the request was successful \n",
    "# other status codes indicate an error and can be seen here: https://en.wikipedia.org/wiki/List_of_HTTP_status_codes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 📥You've Got Response!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"Utility.Number\",\"Utility.Name\",\"Utility.State\",\"Utility.Type\",\"Demand.Summer Peak\",\"Demand.Winter P\n"
     ]
    }
   ],
   "source": [
    "# additionally, we can check the response.text for a preview of the data\n",
    "print(response.text [:100])\n",
    "\n",
    "# the response.text is displayed in a string format, of which we see the first 100 characters with the [:100] notation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So let's break down what we've done so far. To begin, we've now reached out to the dataset hosting site, through the variable URL, which contains the dataset that we would like to work with.\n",
    "This response will be what we work with to save our csv file to the local system.\n",
    "\n",
    "--\n",
    "\n",
    "We also take a moment to view the `response.status_code`, an attribute of our `request.get()` class object. Here, we've assigned this as a variable, `response` before moving in to preview the `response.text` attribute."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have the dataset in memory now as `response`, waiting to be saved to our local OS. Let's do that next by importing a python library `os`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Taylor\\Documents\n"
     ]
    }
   ],
   "source": [
    "import os # We don't need to install `os`, as it's a built-in library in Python\n",
    "\n",
    "# We can set our working directory now, so that we can save the data we accessed to our computer\n",
    "\n",
    "working_directory = os.getcwd()\n",
    "print(working_directory)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: You'll see above my `Current Working Directory` as it was, also known by `cwd`, as the `jupyter cells` have been `executed` in this notebook without `clearing the output` for ease of the reader's _**experience**_. \n",
    "Challenge yourself to change the directory in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Taylor\\Documents\n"
     ]
    }
   ],
   "source": [
    "# This is that code block where you can test changing the cwd, I'm on Windows so I use backslashes\n",
    "working_directory=\"C:\\\\Users\\\\Taylor\\\\Documents\" # We set the working directory VARIABLE to the User's Documents folder\n",
    "os.chdir(working_directory)                      # We change the working directory to the User's Documents folder WITH the os.chdir() FUNCTION\n",
    "print(working_directory)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a new folder in our working directory to store the **data** we accessed from our **requested url**.\n",
    "\n",
    "We can do this with the `os.mkdir()` function, which creates a new folder in the cwd, I'll call it `electricity_data` and wrap it in a `try/except block`, so that if the folder already exists, _we can handle an expected case error_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Taylor\\Documents\n",
      "Folder already exists: electricity_data\n"
     ]
    }
   ],
   "source": [
    "folder_name = \"electricity_data\"        # We set the folder_name VARIABLE to contain the STRING name of the folder we want to create\n",
    "root_directory = os.getcwd()            # Let's rename the working directory to root_directory, this will make sense in a moment\n",
    "print(root_directory)                   # We print the root_directory to see what it is, while print is not necessary, it's good practice to check your code as you go\n",
    "os.chdir(root_directory)                # We change the working directory to the User's Documents folder WITH the os.chdir() FUNCTION\n",
    "try:\n",
    "    os.mkdir(folder_name)\n",
    "    print(\"Folder created successfully: {}\".format(folder_name))\n",
    "except FileExistsError:\n",
    "    print(\"Folder already exists: {}\".format(folder_name))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's join our `root_directory` VARIABLE to the folder VARIABLE we've created. <br>\n",
    "<br>\n",
    "We'll use the `os.path.join()` FUNCTION to do this, this FUNCTION takes the `root_directory` VARIABLE and the `folder_name` VARIABLE as ARGUMENTS, then returns a STRING datatype which we store as, yup, a VARIABLE named `working_directory`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Taylor\\Documents\\electricity_data\n",
      "<class 'str'>\n"
     ]
    }
   ],
   "source": [
    "working_directory = os.path.join(root_directory, folder_name) # working_directory VARIABLE is now the new folder\n",
    "print(working_directory)                                      # Let's print the working_directory VARIABLE showing the changed directory\n",
    "print(type(working_directory))                                # Let's print the type of the working_directory VARIABLE showing it's a STRING"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, you're thinking. <br>\n",
    "So all this, and we accomplished...making a new folder, where did I go wrong in ever learning to right-click? Right? Wrong, is correct. Not today, at least; nope. What? No Worries. Now. <br> \n",
    "Let's combine the dataset we took sitting in RAM or local cache or something. \n",
    "- [x] I'm learning too, yeah? <br>\n",
    "<br>\n",
    "Combined with our folder path:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File created: C:\\Users\\Taylor\\Documents\\electricity_data\\electricity.csv\n"
     ]
    }
   ],
   "source": [
    "file_name=\"electricity.csv\"                                   # We set the file_name VARIABLE to contain the STRING name of the file we want to create\n",
    "file_path = os.path.join(working_directory, file_name)        # We set the file_path VARIABLE to contain the STRING path of the file we want to create\n",
    "with open((file_path), \"wb\") as f:                            # The \"wb\" argument is used to write the file in binary mode. More in the Deep Dives section below.\n",
    "    f.write(response.content)                                 # We use the .content attribute of the response object to write the response.text to the file regardless of unicode\n",
    "    f.close()\n",
    "    \n",
    "print(\"File created: \" + file_path)                           # We print the STRING \"File created: \" and the file_path VARIABLE to the console\n",
    "    \n",
    "# LEARN MORE\n",
    "# Feeling like wtf?   This code block will write the response.text to a file in the working_directory\n",
    "# Ohhhhh. Wait..      The `with` keyword is used to open a file, and the `as` keyword is used to assign the file to a VARIABLE, in our case, f\n",
    "# What about the..?   The `open()` function opens the file, and the \"wb\" argument is used to write the file in binary mode\n",
    "\n",
    "# Honestly, this is quite a bit of code already. \n",
    "# Take a moment to review what we've done thus far. Then, go for the deep dive below, or skip to the Pandas portion of the lesson below.\n",
    "\n",
    "# DEEP DIVES\n",
    "# The `f.write()` function writes the response.text to the file assigned to the variable, in our case, f; \n",
    "#       which is the working_directory; \n",
    "#       which is a STRING; \n",
    "#       set to the User's Documents folder; \n",
    "#       or C:\\Users\\Taylor\\Documents\\electricity_data\n",
    "# While this may seem just, overboard, I want to take this opportunity to highlight the complexity of the code we're writing thus far.\n",
    "\n",
    "# The `f.close()` function closes the file, this is important when the file is needed for other processes to use.\n",
    "# And for those who are asking, what I learned is the end of the `with` block closes the file.\n",
    "# That said, it's your responsibility to close the file if you don't use the `with` block to open the file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you didn't read the comments above, well, I don't blame you.\n",
    "Moving on from those for now,<br>\n",
    "# WE DID IT 🎊 or you did it, the thing did the thing 🥳!\n",
    "\n",
    "You saved a corgi to your mainframe motherboard box attached from the fabric-connected, wire-entangled, mess that we all love. Fin.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### but like for real now what?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Ok, ok. Back on track.** Let's open up this Corgi Dataset with Pandas, a Python Library. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">###### Again for the technicals, Polars is around the corner I know. JUST LET PEOPLE HAVE FU.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pandas in c:\\users\\taylor\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python39\\site-packages (2.0.3)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in c:\\users\\taylor\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python39\\site-packages (from pandas) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in c:\\users\\taylor\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python39\\site-packages (from pandas) (2023.3)\n",
      "Requirement already satisfied: numpy>=1.20.3 in c:\\users\\taylor\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python39\\site-packages (from pandas) (1.25.0)\n",
      "Requirement already satisfied: tzdata>=2022.1 in c:\\users\\taylor\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python39\\site-packages (from pandas) (2023.3)\n",
      "Requirement already satisfied: six>=1.5 in c:\\users\\taylor\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python39\\site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: You are using pip version 22.0.4; however, version 23.2.1 is available.\n",
      "You should consider upgrading via the 'C:\\Users\\Taylor\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\python.exe -m pip install --upgrade pip' command.\n"
     ]
    }
   ],
   "source": [
    "!pip install pandas\n",
    "import pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With that done, let's take a look at our CSV we saved before."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can utilize all the same variables we have saved, such as `working_directory` and, ..actually, that's all we need!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   Utility.Number              Utility.Name Utility.State Utility.Type  \\\n",
      "0              34  City of Abbeville - (SC)            SC    Municipal   \n",
      "1              55   City of Aberdeen - (MS)            MS    Municipal   \n",
      "2              59  City of Abbeville - (LA)            LA    Municipal   \n",
      "3              84       A & N Electric Coop            VA  Cooperative   \n",
      "4              87        City of Ada - (MN)            MN    Municipal   \n",
      "\n",
      "   Demand.Summer Peak  Demand.Winter Peak  Sources.Generation  \\\n",
      "0                13.7                10.8              7000.0   \n",
      "1                32.4                30.3                 0.0   \n",
      "2                28.9                22.0                 0.0   \n",
      "3               154.0               162.4               596.0   \n",
      "4                 2.1                 2.2                 0.0   \n",
      "\n",
      "   Sources.Purchased  Sources.Other  Sources.Total  ...  \\\n",
      "0            59000.0            0.0        66000.0  ...   \n",
      "1           209454.0            0.0       209454.0  ...   \n",
      "2           137264.0            0.0       137264.0  ...   \n",
      "3           743457.0            0.0       744053.0  ...   \n",
      "4            20028.0            0.0        20028.0  ...   \n",
      "\n",
      "   Retail.Commercial.Customers  Retail.Industrial.Revenue  \\\n",
      "0                        460.0                        0.0   \n",
      "1                        662.0                     5638.0   \n",
      "2                        887.0                     3011.1   \n",
      "3                       4227.0                    15516.0   \n",
      "4                        255.0                      190.0   \n",
      "\n",
      "   Retail.Industrial.Sales  Retail.Industrial.Customers  \\\n",
      "0                      0.0                          0.0   \n",
      "1                 120537.0                          1.0   \n",
      "2                  35881.0                         27.0   \n",
      "3                 176162.0                          8.0   \n",
      "4                   2615.0                         58.0   \n",
      "\n",
      "   Retail.Transportation.Revenue  Retail.Transportation.Sales  \\\n",
      "0                            0.0                          0.0   \n",
      "1                            0.0                          0.0   \n",
      "2                            0.0                          0.0   \n",
      "3                            0.0                          0.0   \n",
      "4                            0.0                          0.0   \n",
      "\n",
      "   Retail.Transportation.Customers  Retail.Total.Revenue  Retail.Total.Sales  \\\n",
      "0                              0.0                7536.0             58000.0   \n",
      "1                              0.0               14797.0            204261.0   \n",
      "2                              0.0               12383.0            127579.0   \n",
      "3                              0.0               78507.0            704010.0   \n",
      "4                              0.0                1593.0             20028.0   \n",
      "\n",
      "   Retail.Total.Customers  \n",
      "0                  3844.0  \n",
      "1                  3229.0  \n",
      "2                  5494.0  \n",
      "3                 35934.0  \n",
      "4                  1185.0  \n",
      "\n",
      "[5 rows x 38 columns]\n"
     ]
    }
   ],
   "source": [
    "electricity_dataframe = pandas.read_csv(file_path)\n",
    "print(electricity_dataframe.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that's magical.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But it isn't. And it doesn't have to be. Gain experience and level up as this series continues."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}