pinn_wearables_part_2_data_preprocessing_(1).ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/NSPFD/quantum-computing/blob/main/pinn_wearables_part_2_data_preprocessing_(1).ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1d303bb1-90fc-4a75-a6e8-3c010c3af49b",
      "metadata": {
        "id": "1d303bb1-90fc-4a75-a6e8-3c010c3af49b"
      },
      "source": [
        "## Goal"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "83492d9c-111a-4307-9304-fb8f7557078e",
      "metadata": {
        "id": "83492d9c-111a-4307-9304-fb8f7557078e"
      },
      "source": [
        "We want to first pre-process the dataset to eliminate any redundancy so that we can focus our attention on a useful and simplified dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "94d25227-8b04-41a5-8118-4cbd06b4ea4d",
      "metadata": {
        "id": "94d25227-8b04-41a5-8118-4cbd06b4ea4d"
      },
      "source": [
        "## Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6a44aafb-0d6e-4d2f-baf7-98a4c8510ecc",
      "metadata": {
        "id": "6a44aafb-0d6e-4d2f-baf7-98a4c8510ecc"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0b773209-06d7-4ac6-a09e-1439483391a6",
      "metadata": {
        "id": "0b773209-06d7-4ac6-a09e-1439483391a6"
      },
      "source": [
        "## Load the file"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "601c1710-40fd-4e4b-93c7-b55be1325098",
      "metadata": {
        "id": "601c1710-40fd-4e4b-93c7-b55be1325098"
      },
      "outputs": [],
      "source": [
        "file_path = 'imu_data.log'\n",
        "with open(file_path, 'r') as file:\n",
        "    data = file.readlines()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5c78604b-0000-494e-89a4-aaf31956cba8",
      "metadata": {
        "id": "5c78604b-0000-494e-89a4-aaf31956cba8"
      },
      "source": [
        "## What does the data look like?"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cadd249c-7c01-4993-b795-1fece0b1ab9b",
      "metadata": {
        "id": "cadd249c-7c01-4993-b795-1fece0b1ab9b"
      },
      "source": [
        "![image.png](attachment:47377309-496d-4de8-8b7e-769b31f11bb4.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "10f0eace-57b6-48ae-8014-ae5578812ebc",
      "metadata": {
        "id": "10f0eace-57b6-48ae-8014-ae5578812ebc"
      },
      "source": [
        "## Locate where the actual data starts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c3c93d29-62d5-46be-a0dc-e79a59c68133",
      "metadata": {
        "id": "c3c93d29-62d5-46be-a0dc-e79a59c68133"
      },
      "outputs": [],
      "source": [
        "data_start_index = None\n",
        "for index, line in enumerate(data):\n",
        "    if not line.startswith('//'):\n",
        "        data_start_index = index\n",
        "        break"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e363e604-3d83-44f5-ac13-cb353e7581b6",
      "metadata": {
        "id": "e363e604-3d83-44f5-ac13-cb353e7581b6"
      },
      "source": [
        "## Read the data into a DataFrame skipping the initial metadata line"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b4b6b2a0-bda4-487c-bb40-0082746823ec",
      "metadata": {
        "id": "b4b6b2a0-bda4-487c-bb40-0082746823ec"
      },
      "outputs": [],
      "source": [
        "df = pd.read_csv(file_path, skiprows=data_start_index)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "572105f7-0661-4040-ae36-37ffd30f87ca",
      "metadata": {
        "id": "572105f7-0661-4040-ae36-37ffd30f87ca"
      },
      "source": [
        "1. Pandas is a powerful Python library for data wrangling\n",
        "2. DataFrames are data structures that are similar to excel spreadsheets"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "48f5915e-4698-480e-89fe-1bb9a75ebca0",
      "metadata": {
        "id": "48f5915e-4698-480e-89fe-1bb9a75ebca0"
      },
      "source": [
        "## Investigate timestamps"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d5fe6521-55b5-4b04-8c5e-2e74c4ef0069",
      "metadata": {
        "id": "d5fe6521-55b5-4b04-8c5e-2e74c4ef0069",
        "outputId": "0f642ce0-8ac3-4a4b-a5b5-c3982abd4dc9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Timestamps are in the correct order...\n",
            "Sampling rate is fixed...\n",
            "Sampling interval dt: 2.5e-03 seconds\n",
            "----------------------------------\n"
          ]
        }
      ],
      "source": [
        "t_array = np.array(df['SampleTimeFine'].values)\n",
        "dt_array = 10**-4 * np.diff(t_array)\n",
        "if all(dt_array > 0):\n",
        "    print(\"Timestamps are in the correct order...\")\n",
        "dt_unique_array = np.unique(dt_array)\n",
        "len_dt_unique = len(dt_unique_array)\n",
        "if len_dt_unique == 1:\n",
        "    print(\"Sampling rate is fixed...\")\n",
        "dt = dt_unique_array.item()\n",
        "print(f\"Sampling interval dt: {dt:.1e} seconds\")\n",
        "print(\"----------------------------------\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2a5d4410-2a87-4e0c-a024-8fcb45e5d67a",
      "metadata": {
        "id": "2a5d4410-2a87-4e0c-a024-8fcb45e5d67a"
      },
      "source": [
        "## Check if velocity increments are redundant"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "eac6937c-f3f1-4709-82ef-5b6caedd207c",
      "metadata": {
        "id": "eac6937c-f3f1-4709-82ef-5b6caedd207c",
        "outputId": "181e2744-d742-4efe-ed9d-f37897048798"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "vel_x_checks, max: 2.1e-04, mean: 4.9e-06, min: 0.0e+00\n",
            "vel_y_checks, max: 1.8e-04, mean: 4.7e-06, min: 0.0e+00\n",
            "vel_z_checks, max: 1.5e-04, mean: 5.5e-06, min: 0.0e+00\n",
            "Velocity increment data is redundant...\n",
            "----------------------------------\n"
          ]
        }
      ],
      "source": [
        "vel_x_check_arr = np.abs(np.array(df['Acc_X']*dt - df['VelInc_X']))\n",
        "vel_x_check_max, vel_x_check_mean, vel_x_check_min = vel_x_check_arr.max(), vel_x_check_arr.mean(), vel_x_check_arr.min()\n",
        "print(f\"vel_x_checks, max: {vel_x_check_max:.1e}, mean: {vel_x_check_mean:.1e}, min: {vel_x_check_min:.1e}\")\n",
        "\n",
        "vel_y_check_arr = np.abs(np.array(df['Acc_Y']*dt - df['VelInc_Y']))\n",
        "vel_y_check_max, vel_y_check_mean, vel_y_check_min = vel_y_check_arr.max(), vel_y_check_arr.mean(), vel_y_check_arr.min()\n",
        "print(f\"vel_y_checks, max: {vel_y_check_max:.1e}, mean: {vel_y_check_mean:.1e}, min: {vel_y_check_min:.1e}\")\n",
        "\n",
        "vel_z_check_arr = np.abs(np.array(df['Acc_Z']*dt - df['VelInc_Z']))\n",
        "vel_z_check_max, vel_z_check_mean, vel_z_check_min = vel_z_check_arr.max(), vel_z_check_arr.mean(), vel_z_check_arr.min()\n",
        "print(f\"vel_z_checks, max: {vel_z_check_max:.1e}, mean: {vel_z_check_mean:.1e}, min: {vel_z_check_min:.1e}\")\n",
        "\n",
        "print(\"Velocity increment data is redundant...\")\n",
        "print(\"----------------------------------\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0adf9475-f43e-467d-b592-18db41ba79c9",
      "metadata": {
        "id": "0adf9475-f43e-467d-b592-18db41ba79c9"
      },
      "source": [
        "## Check if orientation increments are redundant"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7f40f315-123f-4415-b234-5e2155fda935",
      "metadata": {
        "id": "7f40f315-123f-4415-b234-5e2155fda935",
        "outputId": "8b1afcab-b4af-40fa-ac60-6fdd4268fcbf"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "ori_x_checks, max: 5.3e-07, mean: 2.5e-07, min: 0.0e+00\n",
            "ori_y_checks, max: 5.4e-07, mean: 2.5e-07, min: 0.0e+00\n",
            "ori_z_checks, max: 5.1e-07, mean: 2.5e-07, min: 0.0e+00\n",
            "Orientation increment data is redundant...\n",
            "----------------------------------\n"
          ]
        }
      ],
      "source": [
        "ori_x_check_arr = np.abs(np.array(df['Gyr_X']*(dt/2) - df['OriInc_q1']))\n",
        "ori_x_check_max, ori_x_check_mean, ori_x_check_min = ori_x_check_arr.max(), ori_x_check_arr.mean(), ori_x_check_arr.min()\n",
        "print(f\"ori_x_checks, max: {ori_x_check_max:.1e}, mean: {ori_x_check_mean:.1e}, min: {ori_x_check_min:.1e}\")\n",
        "\n",
        "ori_y_check_arr = np.abs(np.array(df['Gyr_Y']*(dt/2) - df['OriInc_q2']))\n",
        "ori_y_check_max, ori_y_check_mean, ori_y_check_min = ori_y_check_arr.max(), ori_y_check_arr.mean(), ori_y_check_arr.min()\n",
        "print(f\"ori_y_checks, max: {ori_y_check_max:.1e}, mean: {ori_y_check_mean:.1e}, min: {ori_y_check_min:.1e}\")\n",
        "\n",
        "ori_z_check_arr = np.abs(np.array(df['Gyr_Z']*(dt/2) - df['OriInc_q3']))\n",
        "ori_z_check_max, ori_z_check_mean, ori_z_check_min = ori_z_check_arr.max(), ori_z_check_arr.mean(), ori_z_check_arr.min()\n",
        "print(f\"ori_z_checks, max: {ori_z_check_max:.1e}, mean: {ori_z_check_mean:.1e}, min: {ori_z_check_min:.1e}\")\n",
        "\n",
        "print(\"Orientation increment data is redundant...\")\n",
        "print(\"----------------------------------\")"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.7"
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}