001_chap1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 01: Preliminaries "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why Python for Data Analysis (Data Science)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "> One of the most important languages for data science, machine learning, and general software development in academia and industry."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solving the “Two-Language” Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Python is a suitable language not only for doing research and prototyping but also for building the production systems (also Julia programming language)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What Kinds of Data? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](./images/structured_data2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Essential Python Libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- NumPy ( Numerical Python) \n",
    "- Pandas\n",
    "- Matplotlib\n",
    "- Ipython and Jupyter\n",
    "- Scipy\n",
    "- Sklearn\n",
    "- StatsModel\n",
    "  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Numpy "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Short for Numerical Python, has long been a cornerstone of numerical computing in Python. It provides the data structures, algorithms, and library glue needed for most scientific applications involving numerical data in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> pandas provides high-level data structures and functions designed to make working with structured or tabular data intuitive and flexible. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- It provides convenient indexing functionality to enable you to reshape, slice and dice, perform aggregations, and select subsets of data. Since data manipulation, preparation, and cleaning is such an important skill in data analysis,\n",
    "\n",
    "- [See R vs Pandas comparison](https://pandas.pydata.org/docs/getting_started/comparison/comparison_with_r.html)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> is the most popular Python library for producing plots and other two-dimensional data visualizations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IPython and Jupyter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-  The IPython system can now be used as a kernel (a programming language mode) for using Python with Jupyter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SciPy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> SciPy is a collection of packages addressing a number of foundational problems in scientific computing.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scikit-learn\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> general-purpose machine learning toolkit for Python programmers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Statsmodels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> is a statistical analysis package "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Compared with scikit-learn, statsmodels contains algorithms for classical (primarily frequentist) statistics and econometrics. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other Packages\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-  TensorFlow or PyTorch or Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing Necessary Packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> We can install Python packages using \"Pip\" or \"Conda\". Read more about [pip vs python](https://pythonspeed.com/articles/conda-vs-pip/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The author recommends: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Miniconda, a minimal installation of the conda package manager, along with conda-forge, a community-maintained software distribution based on conda. \n",
    "\n",
    "- This book uses Python 3.10 throughout. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mini-conda"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Conda is a packaging tool and installer that aims to do more than what pip does; handle library dependencies outside of the Python packages as well as the Python packages themselves. Conda also creates a virtual environment, like virtualenv does"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](./images/conda_vs_pip.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mini-forge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- miniforge is the community (conda-forge) driven minimalistic conda installer. Subsequent package installations come thus from conda-forge channel. [Mini-forge](https://github.com/conda-forge/miniforge)\n",
    "\n",
    "- miniconda is the Anaconda (company) driven minimalistic conda installer. Subsequent package installations come from the anaconda channels (default or otherwise).\n",
    "\n",
    "-  miniforge started because miniconda doens't support aarch64, very quickly the 'PyPy' people jumped on board, and in the mean time there are also miniforge versions for all Linux architectures, as well as MacOS.\n",
    "\n",
    "- AARCH64, sometimes also referred to as ARM64, is a CPU architecture developed by ARM Ltd., and a 64-bit extension of the pre-existing ARM architecture. ARM architectures are primarily known for their energy efficiency and low power consumption. For that reason, virtually all mobile phones and tablets today use ARM architecture-based CPUs.\n",
    "\n",
    "- Although AARCH64 and x64 (Intel, AMD, …) are both 64-bit CPU architectures, their inner basics are vastly different. Programs compiled for one platform, won’t work on the other (except with some magic), and vice-versa. That means, software does not only need to be recompiled, but often requires extensive optimization for either platform."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The first step is to configure conda-forge as your default package channel by running the following commands in a shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning: 'conda-forge' already in 'channels' list, moving to the top\n"
     ]
    }
   ],
   "source": [
    "! conda config --add channels conda-forge\n",
    "! conda config --set channel_priority strict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will install the essential packages used throughout the book (along with their dependencies) with conda install"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conda create -y -n pydata-book python=3.10 # create enviroment with python 3.10 installed\n",
    "conda activate pydata-book # activate enviroment \n",
    "(pydata-book) $ conda install -y pandas jupyter matplotlib # install a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install complete packages used in the the book"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "conda install lxml beautifulsoup4 html5lib openpyxl \\\n",
    "               requests sqlalchemy seaborn scipy statsmodels \\\n",
    "               patsy scikit-learn pyarrow pytables numba"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Should I use Pip or Conda ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> While you can use both conda and pip to install packages, you should avoid updating packages originally installed with conda using pip (and vice versa), as doing so can lead to environment problems. I recommend sticking to conda if you can and falling back on pip only for packages which are unavailable with conda install."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> conda install should always be preferred, but some packages are not available through conda so if conda install $package_name fails, try pip install $package_name."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What can we do with Conda? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Many commands : create env, activate env, delete env, lists env\n",
    "  \n",
    "- Install tldr (https://github.com/tldr-pages/tldr) : The tldr-pages project is a collection of community-maintained help pages for command-line tools, that aims to be a simpler, more approachable complement to traditional\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Navigating This Book"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Chapter two and three: provides prerequisite knowledge for the remainder of the book. If you have Python experienc you can skip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Chapter four : Numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Chapter five : Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Chapter six : Data loading, Storage and File format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Chapter seven : Data cleaning and Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Chpater eight : Data wrangling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Chapter Nine : Plotting and Visualization\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Chapter 10 : Data agreegation and Group operation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Conventions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The Python community has adopted a number of naming conventions for commonly used modules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (1580968464.py, line 5)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  Input \u001b[0;32mIn [1]\u001b[0;36m\u001b[0m\n\u001b[0;31m    import statsmodels as\u001b[0m\n\u001b[0m                         ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import statsmodels as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> This means that when you see np.arange, this is a reference to the arange function in NumPy. This is done because it’s considered bad practice in Python software development to import everything (from numpy import *) from a large package like NumPy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "\n",
    "x = np.random.random((64, 3, 32, 10)) \n",
    "y = np.random.random((32, 10)) \n",
    "\n",
    "z = np.maximum(x, y)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.12",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "d203a7fbe37afbb990fedfc21c321928443618f3d7b991e0237ff71906aa031f"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}