xgboost-workflow-2-train.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## XGBoost Complete Project Workflow in Amazon SageMaker\n",
    "### Model Training\n",
    "    \n",
    "1. [Local Mode training](#LocalModeTraining)\n",
    "2. [SageMaker hosted training](#SageMakerHostedTraining)\n",
    "3. [Automatic model tuning](#AutomaticModelTuning)\n",
    "\n",
    "##  Local Mode training<a class=\"anchor\" id=\"LocalModeTraining\">\n",
    "Local Mode in Amazon SageMaker is a convenient way to make sure your code is working locally as expected before moving on to full scale, hosted training in a separate, more powerful SageMaker-managed cluster. To train in Local Mode, it is necessary to have docker-compose or nvidia-docker-compose (for GPU instances) installed. Running the following commands will install docker-compose or nvidia-docker-compose, and configure the notebook environment for you.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Amazon SageMaker supports two ways to use the XGBoost algorithm:\n",
    "\n",
    "- XGBoost built-in algorithm\n",
    "- XGBoost open source algorithm\n",
    "\n",
    "The XGBoost open source algorithm provides the following benefits over the built-in algorithm:\n",
    "\n",
    "- Latest version - The open source XGBoost algorithm typically supports a more recent version of XGBoost. To see the XGBoost version that is currently supported, see XGBoost SageMaker Estimators and Models.\n",
    "- Flexibility - Take advantage of the full range of XGBoost functionality, such as cross-validation support. You can add custom pre- and post-processing logic and run additional code after training.\n",
    "- Scalability - The XGBoost open source algorithm has a more efficient implementation of distributed training, which enables it to scale out to more instances and reduce out-of-memory errors.\n",
    "- Extensibility - Because the open source XGBoost container is open source, you can extend the container to install additional libraries and change the version of XGBoost that the container uses. For an example notebook that shows how to extend SageMaker containers, see Extending our PyTorch containers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we'll import the variables stored from previous notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import sagemaker\n",
    "from parameter_store import ParameterStore\n",
    "\n",
    "ps = ParameterStore()\n",
    "parameters = ps.read()\n",
    "\n",
    "bucket = parameters['bucket']\n",
    "s3_prefix = parameters['s3_prefix']\n",
    "raw_s3 = parameters['raw_s3']\n",
    "train_dir = parameters['train_dir']\n",
    "test_dir = parameters['test_dir']\n",
    "train_dir_csv = parameters['train_dir_csv']\n",
    "test_dir_csv = parameters['test_dir_csv']\n",
    "role = parameters['role']\n",
    "sess = sagemaker.Session()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget -q https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-script-mode/master/local_mode_setup.sh\n",
    "!wget -q https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-script-mode/master/daemon.json    \n",
    "!/bin/bash ./local_mode_setup.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll set up a XGBoost Estimator for Local Mode training. Key parameters for the Estimator include:\n",
    "\n",
    "- `train_instance_type`: the kind of hardware on which training will run. In the case of Local Mode, we simply set this parameter to `local` to invoke Local Mode training on the CPU, or to `local_gpu` if the instance has a GPU.  \n",
    "- The algorithm’s hyperparameters, which are passed in as a dictionary. \n",
    "\n",
    "Recall that we are using Local Mode here mainly to make sure our code is working. Accordingly, instead of performing a full cycle of training with many epochs (passes over the full dataset), we'll train only for a small number of epochs just to confirm the code is working properly and avoid wasting full-scale training time unnecessarily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "x_train = np.load(f'{train_dir}/x_train.npy')\n",
    "y_train = np.load(f'{train_dir}/y_train.npy')\n",
    "x_test = np.load(f'{test_dir}/x_test.npy')\n",
    "y_test = np.load(f'{test_dir}/y_test.npy')\n",
    "\n",
    "train_df = pd.DataFrame(data=x_train)\n",
    "train_df['target'] = y_train\n",
    "first_col = train_df.pop('target')\n",
    "train_df.insert(0, 'target', first_col)\n",
    "\n",
    "test_df = pd.DataFrame(data=x_test)\n",
    "test_df['target'] = y_test\n",
    "first_col = test_df.pop('target')\n",
    "test_df.insert(0, 'target', first_col)\n",
    "\n",
    "train_df.to_csv('./data/train_csv/train.csv', header=False, index=False)\n",
    "test_df.to_csv('./data/test_csv/test.csv', header=False, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.xgboost import XGBoost\n",
    "\n",
    "train_instance_type = 'local'\n",
    "hyperparameters = {'num_round': 6}\n",
    "\n",
    "local_estimator_parameters = {'entry_point':'train_deploy.py',\n",
    "                              'train_instance_type' : train_instance_type,\n",
    "                              'train_instance_count': 1,\n",
    "                              'hyperparameters': hyperparameters,\n",
    "                              'role' : role,\n",
    "                              'base_job_name':'xgboost-local-model',\n",
    "                              'framework_version':'1.0-1',\n",
    "                              'py_version':'py3'}\n",
    "\n",
    "local_estimator = XGBoost(**local_estimator_parameters)\n",
    "inputs = {'train': f'file://{train_dir_csv}',\n",
    "          'test': f'file://{test_dir_csv}'}\n",
    "local_estimator.fit(inputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "local_model_data = local_estimator.model_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Experiment tracking <a class=\"anchor\" id=\"Experiment\">\n",
    "SageMaker experiments can track all the model training iterations. Experiments are a great way to organize your data science work. You can create experiments to organize all your model development work for : [1] a business use case you are addressing (e.g. create experiment named “customer churn prediction”), or [2] a data science team that owns the experiment (e.g. create experiment named “marketing analytics experiment”), or [3] a specific data science and ML project. Think of it as a “folder” for organizing your “files”."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "!{sys.executable} -m pip install sagemaker-experiments\n",
    "!pip install --upgrade boto3==1.16.27\n",
    "!pip install sagemaker==1.72.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from smexperiments.experiment import Experiment\n",
    "from time import gmtime, strftime\n",
    "import boto3\n",
    "\n",
    "sm = boto3.client('sagemaker')\n",
    "\n",
    "xgboost_experiment = Experiment.create(\n",
    "    experiment_name=\"boston-housing-regression-{}\".format(strftime(\"%d-%H-%M-%S\", gmtime())), \n",
    "    description=\"Boston housing price estimation.\", \n",
    "    sagemaker_boto_client=sm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  SageMaker hosted training <a class=\"anchor\" id=\"SageMakerHostedTraining\">\n",
    "\n",
    "Now that we've confirmed our code is working locally, we can move on to use SageMaker's hosted training functionality. Hosted training is preferred for doing actual training, especially large-scale, distributed training.  Unlike Local Mode training, for hosted training the actual training itself occurs not on the notebook instance, but on a separate cluster of machines managed by SageMaker.  Before starting hosted training, the data must be in S3, or an EFS or FSx for Lustre file system. We uploaded to S3 in the previous notebook, so we're good to go here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upload to S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.session import s3_input\n",
    "import os\n",
    "\n",
    "boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(s3_prefix, 'data/train/train.csv')).upload_file('./data/train_csv/train.csv')\n",
    "boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(s3_prefix, 'data/test/test.csv')).upload_file('./data/test_csv/test.csv')\n",
    "s3_input_train_uri = f's3://{bucket}/{s3_prefix}/data/train/train.csv'\n",
    "s3_input_test_uri = f's3://{bucket}/{s3_prefix}/data/test/test.csv'\n",
    "s3_input_train = s3_input(s3_input_train_uri, content_type='csv')\n",
    "s3_input_test = s3_input(s3_input_test_uri, content_type='csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create the XGBoost model and pass in the required hyperparameter. Details of hyperparamters can be found [here](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost_hyperparameters.html).\n",
    "\n",
    "We're now ready to set up an Estimator object for hosted training. It is similar to the Local Mode Estimator, except the `train_instance_type` has been set to a SageMaker ML instance type instead of `local` for Local Mode. Also, since we know our code is working now, we'll train for a larger number of epochs with the expectation that model training will converge to an improved, lower validation loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "from sagemaker.estimator import Estimator\n",
    "\n",
    "container = get_image_uri(sess.boto_region_name, 'xgboost')\n",
    "train_instance_type = 'ml.m4.xlarge'\n",
    "hyperparameters = {'num_round': 8}\n",
    "estimator = Estimator(container,\n",
    "                      role=role,\n",
    "                      train_instance_type=train_instance_type,\n",
    "                      train_instance_count=1,\n",
    "                      hyperparameters=hyperparameters,\n",
    "                      base_job_name='xgboost-hosted-model')\n",
    "\n",
    "inputs = {'train': s3_input_train, 'validation': s3_input_test}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After starting the hosted training job with the `fit` method call below, you should observe the training converge over the longer number of epochs to a validation loss that is considerably lower than that which was achieved in the shorter Local Mode training job.  Can we do better? We'll look into a way to do so in the **Automatic Model Tuning** section below. \n",
    "\n",
    "We will add the experiment name to the `fit()` call to group this training job as part of our experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment_config={\n",
    "    \"ExperimentName\": xgboost_experiment.experiment_name,\n",
    "    \"TrialComponentDisplayName\": \"Training\",\n",
    "}\n",
    "\n",
    "estimator.fit(inputs, experiment_config=experiment_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that training finished, we can use SageMaker Experiments to examine the results and see how it compares to other training jobs within the experiment. Right now this is the only job captured in Experiments, but let's take a look anyway to see what data it stores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.analytics import ExperimentAnalytics\n",
    "\n",
    "trial_component_analytics = ExperimentAnalytics(\n",
    "    sagemaker_session=sess, \n",
    "    experiment_name=xgboost_experiment.experiment_name,\n",
    "    sort_order=\"Descending\",\n",
    "    metric_names=['validation:rmse']\n",
    ")\n",
    "\n",
    "columns = ['TrialComponentName', 'validation:rmse - Last', 'num_round']\n",
    "trial_component_analytics.dataframe()[columns]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to look at your training job analytics is by using the `sagemaker.analytics.TrainingJobAnalytics` class in the SageMaker Python SDK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.analytics import TrainingJobAnalytics\n",
    "\n",
    "metric_names = ['train:rmse', 'validation:rmse']\n",
    "TrainingJobAnalytics(estimator.latest_training_job.name, metric_names=metric_names).dataframe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_job_name = estimator.latest_training_job.name\n",
    "remote_model_data = estimator.model_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with the Local Mode training, hosted training produces a model saved in S3 that we can retrieve.  This is an example of the modularity of SageMaker: having trained the model in SageMaker, you can now take the model out of SageMaker and run it anywhere else.  Alternatively, you can deploy the model into a production-ready environment using SageMaker's hosted endpoints functionality, as shown in the **SageMaker hosted endpoint** section below.\n",
    "\n",
    "Retrieving the model from S3 is very easy:  the hosted training estimator you created above stores a reference to the model's location in S3.  You simply copy the model from S3 using the estimator's `model_data` property and unzip it to inspect the contents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimator.model_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws s3 cp {estimator.model_data} ./model/model.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The unzipped archive should include the assets required by PyTorch to load the model and serve it:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -xvzf ./model/model.tar.gz -C ./model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Managed Spot Training <a class=\"anchor\" id=\"ManagedSpotTraining\">\n",
    "    \n",
    "In this next example we will create a Hosted Training job with Managed Spot Training and Checkpointing enabled.\n",
    "    \n",
    "The Managed Spot Training Estimator is similar to the Host Training estimator, except we must add the following additional arugments: \n",
    "   * `train_use_spot_instances: True` (Boolean)\n",
    "   * `train_max_run: 1200` (Integer - Seconds)\n",
    "   * `train_max_wait: 2400` (Integer - Seconds)\n",
    "   * `checkpoint_s3_uri: s3://{}/{}/checkpoint\".format(bucket, s3_prefix)` (must match a valid s3 URI)\n",
    "    \n",
    "**Note:** `train_max_wait` must be equal or greater than or equal to `train_max_run` or you will get an arugment exception error.    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperparameters = {'num_round': 8}\n",
    "estimator = Estimator(container,\n",
    "                      role=role,\n",
    "                      train_instance_type=train_instance_type,\n",
    "                      train_instance_count=1,\n",
    "                      hyperparameters=hyperparameters,\n",
    "                      base_job_name='xgboost-hosted-model',\n",
    "                      train_use_spot_instances=True,\n",
    "                      train_max_run=1200,\n",
    "                      train_max_wait=2400,\n",
    "                      checkpoint_s3_uri=f's3://{bucket}/{s3_prefix}/checkpoint')\n",
    "\n",
    "inputs = {'train': s3_input_train, 'validation': s3_input_test}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment_config={\n",
    "    \"ExperimentName\": xgboost_experiment.experiment_name,\n",
    "    \"TrialComponentDisplayName\": \"Training\",\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimator.fit(inputs=inputs,\n",
    "              experiment_config=experiment_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.analytics import ExperimentAnalytics\n",
    "\n",
    "trial_component_analytics = ExperimentAnalytics(\n",
    "    sagemaker_session=sess, \n",
    "    experiment_name=xgboost_experiment.experiment_name,\n",
    "    sort_order=\"Descending\",\n",
    "    metric_names=['val_loss']\n",
    ")\n",
    "\n",
    "trial_component_analytics.dataframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automatic Model Tuning <a class=\"anchor\" id=\"AutomaticModelTuning\">\n",
    "\n",
    "So far we have simply run one Local Mode training job, one Hosted Training job and one Managed Spot Training job without any real attempt to tune hyperparameters to produce a better model, other than increasing the number of epochs.  Selecting the right hyperparameter values to train your model can be difficult, and typically is very time consuming if done manually. The right combination of hyperparameters is dependent on your data and algorithm; some algorithms have many different hyperparameters that can be tweaked; some are very sensitive to the hyperparameter values selected; and most have a non-linear relationship between model fit and hyperparameter values.  SageMaker Automatic Model Tuning helps automate the hyperparameter tuning process:  it runs multiple training jobs with different hyperparameter combinations to find the set with the best model performance.\n",
    "\n",
    "We begin by specifying the hyperparameters we wish to tune, and the range of values over which to tune each one.  We also must specify an objective metric to be optimized:  in this use case, we'd like to minimize the validation loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = {'train': s3_input_train, 'validation': s3_input_test}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tuner import IntegerParameter, ContinuousParameter, HyperparameterTuner\n",
    "\n",
    "hyperparameter_ranges = {\n",
    "  'num_round': IntegerParameter(2, 10),\n",
    "  'alpha': ContinuousParameter(0, 2)\n",
    "}\n",
    "\n",
    "objective_metric_name = 'validation:rmse' #mae, map, auc, error, among others\n",
    "objective_type = 'Minimize'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we specify a HyperparameterTuner object that takes the above definitions as parameters.  Each tuning job must be given a budget:  a maximum number of training jobs.  A tuning job will complete after that many training jobs have been executed.  \n",
    "\n",
    "We also can specify how much parallelism to employ, in this case five jobs, meaning that the tuning job will complete after three series of five jobs in parallel have completed.  For the default Bayesian Optimization tuning strategy used here, the tuning search is informed by the results of previous groups of training jobs, so we don't run all of the jobs in parallel, but rather divide the jobs into groups of parallel jobs.  There is a trade-off: using more parallel jobs will finish tuning sooner, but likely will sacrifice tuning search accuracy. \n",
    "\n",
    "Now we can launch a hyperparameter tuning job by calling the `fit` method of the HyperparameterTuner object.  The tuning job may take around 10 minutes to finish.  While you're waiting, the status of the tuning job, including metadata and results for invidual training jobs within the tuning job, can be checked in the SageMaker console in the **Hyperparameter tuning jobs** panel.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "\n",
    "container = get_image_uri(sess.boto_region_name, 'xgboost')\n",
    "estimator = sagemaker.estimator.Estimator(container,\n",
    "                              role=role,\n",
    "                              train_instance_type='ml.m4.xlarge',\n",
    "                              train_instance_count=1,\n",
    "                              hyperparameters=hyperparameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuner_parameters = {'estimator':estimator,\n",
    "                    'objective_metric_name':objective_metric_name,\n",
    "                    'hyperparameter_ranges':hyperparameter_ranges,\n",
    "                    #'metric_definitions':metric_definitions,\n",
    "                    'objective_type': objective_type,\n",
    "                    'max_jobs':4,\n",
    "                    'max_parallel_jobs':2}\n",
    "\n",
    "tuner = HyperparameterTuner(**tuner_parameters)\n",
    "\n",
    "tuning_job_name = \"xgboost-tuning-{}\".format(strftime(\"%d-%H-%M-%S\", gmtime()))\n",
    "tuner.fit(inputs, job_name=tuning_job_name)\n",
    "tuner.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the tuning job is finished, we can use the `HyperparameterTuningJobAnalytics` object from the SageMaker Python SDK to list the top 5 tuning jobs with the best performance. Although the results vary from tuning job to tuning job, the best validation loss from the tuning job (under the FinalObjectiveValue column) likely will be substantially lower than the validation loss from the hosted training job above, where we did not perform any tuning other than manually increasing the number of epochs once.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "tuner_metrics = sagemaker.HyperparameterTuningJobAnalytics(tuning_job_name)\n",
    "tuner_metrics.dataframe().sort_values(['FinalObjectiveValue'], ascending=True).head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The total training time and training jobs status can be checked with the following lines of code. Because automatic early stopping is by default off, all the training jobs should be completed normally.  For an example of a more in-depth analysis of a tuning job, see the SageMaker official sample [HPO_Analyze_TuningJob_Results.ipynb](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/hyperparameter_tuning/analyze_results/HPO_Analyze_TuningJob_Results.ipynb) notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "total_time = tuner_metrics.dataframe()['TrainingElapsedTimeSeconds'].sum() / 3600\n",
    "print(\"The total training time is {:.2f} hours\".format(total_time))\n",
    "tuner_metrics.dataframe()['TrainingJobStatus'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use the training artifacts created in this notebook in downstream notebooks for model deployment. Store them here for later retrieval."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ps.update({'local_model_data': local_model_data, 'remote_model_data': remote_model_data,\n",
    "           'training_job_name': training_job_name, 'tuning_job_name': tuning_job_name,\n",
    "           's3_input_train_uri': s3_input_train_uri, 's3_input_test_uri': s3_input_test_uri})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ps.store()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}